Commit graph

307 commits

Author SHA1 Message Date
Matt Jankowski
682507bc3c Specs for pubsub subscribe service (#2951)
* Add spec for pubsubhubbub/subscribe

* Refactor pubsubhubbub/subscribe service
2017-05-09 20:48:30 +02:00
Matt Jankowski
441d6dc734 Spec and refactor for pubsubhubbub/unsubscribe service (#2946)
* Add coverage for pubsub unsubscribe service

* Refactor pubsub unsubscribe service
2017-05-09 19:58:18 +02:00
Akihiko Odaki
d5cabfe5c6 Use CLD3 (#2949)
Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.

Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.

CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.

On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.

CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.

The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
2017-05-09 19:58:03 +02:00
abcang
08e94d1b19 Fixed bug that timeline can not be displayed by InvalidURIError (#2947) 2017-05-09 18:17:41 +02:00
Matt Jankowski
5bea42412e Services specs for subscribe and unsubscribe (#2928)
* Add specs for unsubscribe service

* Fix non existent methods in unsubscribe service

* Clean up status handling in subscribe service
2017-05-09 00:45:02 +02:00
Matt Jankowski
04166c4a35 Specs for API push controller, with refactor (#2926)
* Coverage for api push controller

* Refactor the api/push controller
2017-05-09 00:44:30 +02:00
Matt Jankowski
7a7d12d27f Delegate Account#user_locale method and allow nil (#2927) 2017-05-08 23:10:50 +02:00
Matt Jankowski
bba537a7be Improve allowed language handling (#2897)
* Dont allow empty value in user allowed languages

* Sanitize language input to reject blank values in array
2017-05-08 03:32:52 +02:00
Matt Jankowski
b8ba719f73 Unblock domain service specs/refactor (#2867)
* Add spec for unblock domain service

* Refactor UnblockDomainService
2017-05-07 14:44:28 +02:00
alpaca-tc
74036a2c9d Hotfix convert string from symbol (#2856)
* Convert key to string from symbol

* Prefer :public_send instead of
2017-05-06 23:06:52 +02:00
alpaca-tc
05b72368ed Fixes slow query in notifications api (#2851)
* Add inclusion validation for activity_type

* When excluding all types with `WHERE NOT IN`, the query search all rows.
2017-05-06 21:55:40 +02:00
alpaca-tc
59804abc3d Optimize MuteService and AfterBlockService (#2836) 2017-05-06 16:31:07 +02:00
Eugen Rochko
b8e166894b Fix Scheduler::SubscriptionsScheduler (#2834)
* Fix Scheduler::SubscriptionsScheduler, add worker test for it

* Change production log level of Sidekiq to "warn" instead of "info"
2017-05-06 13:05:03 +02:00
Matt Jankowski
3f5b994ff0 Stream entry specs and refactor to use delegate (#2827)
* Add coverage for stream entry delegated methods

* Use delegate with allow_nil to clean up stream entry
2017-05-06 04:00:21 +02:00
Matt Jankowski
484c9709b6 Misc spec coverage improvements (#2821)
* Dont use raise_error by itself (avoids warning)

* Add coverage for AccountFilter

* Improve coverage and refactor for Subscription#lease_seconds

* Improve coverage and refactor for NotificationMailer

* Simplify assignment of min/max threshold on subscription
2017-05-05 14:56:00 -04:00
masarakki
d08f1112d5 use Twitter::Extractor for creating links (#2502) 2017-05-05 19:48:22 +02:00
Matt Jankowski
20c37ed0f9 Add specs (and refactor) of FetchRemoteResourceService and SearchService (#2812)
* Coverage for fetch remote resource service

* Refactor fetch remote resource service

* Coverage for search service

* Refactor search service
2017-05-05 17:26:04 +02:00
Eugen Rochko
81584779cb More robust PuSH subscription refreshes (#2799)
* Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix #2702 - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
2017-05-05 02:23:01 +02:00
Matt Jankowski
f9d398e8fb Remove the react-rails gem (#2801)
* Remove react-rails gem

* Fix broken view spec
2017-05-04 23:45:18 +02:00
alpaca-tc
74c8ca699c Delete records in smaller transaction (#2802) 2017-05-04 23:44:39 +02:00
Matt Jankowski
84eb425f38 Remove redis clear from before in spec (#2795) 2017-05-04 20:53:39 +02:00
Matt Jankowski
2bd132d458 Replace best_in_place editor on admin settings page (#2789)
* Remove best_in_place

* Replace best_in_place usage with rails helpers

* Move admin/settings#index to #edit action

* Remove click_to__edit from i18n
2017-05-04 18:12:44 +02:00
Eugen Rochko
13c16b4e95 Likely fix #2458, fix #2031 - handle out-of-order deletes for statuses (#2734)
* Likely fix #2458, fix #2031 - handle out-of-order deletes for statuses

If a delete arrives before the original status, cache that information
for 6h, and if the original status arrives in that window, ignore it

* Add test case
2017-05-04 04:34:57 +02:00
ThibG
8d4e7504b1 Additional specs for URI handling (#2759) 2017-05-03 20:40:14 +02:00
Eugen Rochko
bafd22ecf4 Fix #2706 - Always respond with 200 to PuSH payloads (#2733)
Fix #2196 - Respond with 201 when Salmon accepted, 400 when unverified
Fix #2629 - Correctly handle confirm_domain? for local accounts
Unify rules for extracting author acct from XML, prefer <email>, fall back
to <name> + <uri> (see also #2017, #2172)
2017-05-03 17:02:18 +02:00
Matt Jankowski
8c5ad23b24 Language improvements, replace whatlanguage with CLD (#2753)
* add failing en specs

* add cld2 gem

* Replace WhatLanguage with CLD
2017-05-03 16:59:31 +02:00
Eugen Rochko
f5bf5ebb82 Replace sprockets/browserify with Webpack (#2617)
* Replace browserify with webpack

* Add react-intl-translations-manager

* Do not minify in development, add offline-plugin for ServiceWorker background cache updates

* Adjust tests and dependencies

* Fix production deployments

* Fix tests

* More optimizations

* Improve travis cache for npm stuff

* Re-run travis

* Add back support for custom.scss as before

* Remove offline-plugin and babili

* Fix issue with Immutable.List().unshift(...values) not working as expected

* Make travis load schema instead of running all migrations in sequence

* Fix missing React import in WarningContainer. Optimize rendering performance by using ImmutablePureComponent instead of
React.PureComponent. ImmutablePureComponent uses Immutable.is() to compare props. Replace dynamic callback bindings in
<UI />

* Add react definitions to places that use JSX

* Add Procfile.dev for running rails, webpack and streaming API at the same time
2017-05-03 02:04:16 +02:00
Matt Jankowski
268dd32d76 Auth sign out (#2511)
* Add a spec for signing out

* Add spec showing that suspended user gets a 403 forbidden on sign out

* Allow suspended account users to sign out
2017-05-02 23:37:58 +02:00
ThibG
bea97ea766 Add rspec to further specify FollowRemoteAccountService (#2414) 2017-05-02 23:37:26 +02:00
Kaylee
7880671f35 Add option to disable two factor auth in admin accounts panel. (#2584)
* Add option to disable two factor auth in admin accounts panel.
Closes #2578

* Add @mjankowski's suggestions.
* Moves destroy actions behind User#disable_two_factor!
* Adds spec coverage for Admin:TwoFactorAuthenticationsController and User#disable_two_factor!
2017-05-02 21:07:12 +02:00
Yamagishi Kazutoshi
6f75c8451d Fix subscription expiration condition (#2715)
* Fix subscription expiration condition

* dry and add spec
2017-05-02 18:21:22 +02:00
Matt Jankowski
a4859446ab Coverage for remote follows (#2694)
* Add coverage for create with empty acct value

* Add coverage for create with webfinger failure

* Add coverage for create with webfinger providing bad values

* Add coverage for create when webfinger is good

* Add coverage for session[:remote_follow] having data

* Simplify how remote follow pulls acct from session

* Remote follow behaves more like model

* Move the discovery portions of remote follow out of controller

* Check for suspended accounts
2017-05-02 00:44:23 +02:00
Matt Jankowski
f025cc6782 Filter on allowed user language preferences (#2361)
* Naive approached to timeline filtering

* Convert allowed_languages into a db column

* Allow users to choose languages to see statuses in

* Style list items as two columns

* Add a hint to explain language filtering preference
2017-05-01 17:42:13 +02:00
Matt Jankowski
5259319cf5 Reports spec for media attachments, clean up method (#2660)
* Add coverage for Report#media_attachments

* Direct query on media attachment
2017-04-30 18:15:49 +02:00
Akihiko Odaki
a6788662b0 Revert "Use PostgreSQL inheritance for blocks and mutes (#2520)" (#2634)
This reverts commit 5135d609b7.
2017-04-30 08:49:24 -04:00
Matt Jankowski
fdcf884cf7 Extract user tracking into concern (#2600) 2017-04-30 00:28:16 +02:00
Akihiko Odaki
5135d609b7 Use PostgreSQL inheritance for blocks and mutes (#2520) 2017-04-30 00:27:31 +02:00
Matt Jankowski
f48cb3eb17 More coverage yes more even more (#2627)
* Add coverage for admin/confirmations controller

* Coverage for statuses controller show action

* Add coverage for admin/domain_blocks controller

* Add coverage for settings/profiles#update
2017-04-30 00:25:38 +02:00
Joël Quenneville
27301312a6 Add tests to the Feed model (#2594)
This adds a test for the `Feed#get` method. While the data
transformations in `Feed#get` may seem redundant, they are important to
maintain the order from Redis. The tests I wrote will fail if someone
tries to refactor away this "redundancy" (as I tried to do in the first
iteration of this change).
2017-04-29 00:21:35 +02:00
Patrick Figel
8ac7fca5d0 Set correct attachment type for rejected media (#2599)
In #2110, a new attachment type "unknown" was introduced for
attachments that were rejected due to a domain being blocked using
reject_media. However, the "type" field was never set to "unknown"
because a default value of "0" (image) is set for that column,
causing the `type.blank?` expression to always equal false.

This version uses type_changed? instead, causing the type to be set
to "unknown" unless a type has been explicitly set. This introduces
a small change in behaviour causing the type to be set to unknown
before paperclip calls `before_post_process`. Presumably this
behaviour is more appropriate than the current one because the
attachment type has not been determined by that point.

Included are new tests for `ProcessFeedService` and
`UpdateRemoteProfileService` which now check that remote media is
downloaded for non-blocked domains and is rejected for others.
2017-04-29 00:18:32 +02:00
Matt Jankowski
a823509b99 Fix broken view spec for about/links (#2591) 2017-04-28 17:54:03 +02:00
Matt Jankowski
9262f6968b Fix broken spec for about/links view partial (#2586) 2017-04-28 16:45:24 +02:00
Yamagishi Kazutoshi
71e73e36cd Fix spec for hide get started (#2585) 2017-04-28 16:44:57 +02:00
Matt Jankowski
9566893cc9 More controller specs (#2561)
* Add render_views in more places

* Delegate methods from account to user with allow nil true, so that admin accounts show view renders when missing a user

* Use actual account instances in authorize follow controller spec
2017-04-28 15:12:37 +02:00
Matt Jankowski
429480bb77 Return missing page when tag does not exist (#2563) 2017-04-28 15:11:21 +02:00
Matt Jankowski
effb08edbb More status specs (#2564)
* Add rough outline of coverage needed for public timeline

* Specs for visibility, replies, boosts

* Specs for silenced account

* Specs for local_only option

* Specs for blocks and mutes

* Add tentative spec around including other silenced account statuses

* Add with_public_visibility scope

* Add simple coverage for tag_timeline

* Tag timeline includes replies

* Replace tag.statuses with a tagged_with scope in tag timeline method

* Use with_public_visibility in tag timeline

* Extract common scope between public and tag timelines to method

* Extract local domain check to local_only scope

* Extract local_only check to starting scope method

* Move list of excluded from timeline account ids to account model

* Simplify excluded accounts list on account model

* Only join accounts when needed

* Rename method for account specific filtering

* Extract method for account exclusions

* Fix bug where silenced accounts were not including statuses from other silenced accounts

* DRY up filter application from account or no account

* timeline_scope can be private

* Add spec showing that account can find its excluded accounts ids

* Add spec which fails if local_only does not have a left outer join

* rubocop
2017-04-28 15:10:41 +02:00
Matt Jankowski
a39e719b39 Add render_views to admin/reports controller spec (#2548) 2017-04-28 01:21:38 +02:00
Matt Jankowski
9736753985 Fix remote follow (#2547)
* Add coverage for remote_follow#new

* Fix reference to authorize_follows/card partial
2017-04-28 00:16:44 +02:00
Eugen Rochko
2af4f3c4e2 Improve shared status verification (#2525)
* Instead of parsing shared status contents verbatim, make roundtrip
to purported original URL. Confirm that the "original" URL is from the
same domain as the author it claims to be from.

* Fix obvious typo, add comment

* Use URI look-up first

* Add test, update Goldfinger dependency to make less useless HTTP requests per Webfinger lookup
2017-04-27 17:06:47 +02:00
Matt Jankowski
b48f2cbc8b Catch error when server decryption fails on 2FA (#2512) 2017-04-27 15:18:21 +02:00