Commit graph

162 commits

Author SHA1 Message Date
David Yip
c2a92dffc9 Add some examples for Glitch::KeywordMute::TagMatcher. #208. 2017-11-15 23:31:49 -06:00
David Yip
08652baab0 Replace =~ with #matches?. #208.
=~ made sense when we were passing it through to a regex, but we're no
longer doing that: TagMatcher looks at individual tags and returns a
value that *looks* like what you get out of #=~ but really isn't that
meaningful.  Probably a good idea to not subvert convention like this
and instead use a name with guessable intent.
2017-11-15 18:27:25 -06:00
David Yip
cb4ef24ac9 Match keyword mute filter on hashtags. #208.
It is reasonable to expect someone to enter #foo to mute hashtag #foo.
However, tags are recorded on statuses without the preceding #.
To adjust for this, we build a separate tag matcher and use
Tag::HASHTAG_RE to extract a hashtag from the hashtag syntax.
2017-11-15 18:12:29 -06:00
beatrix
04508868b0
Merge pull request #212 from aschmitz/feat/mute-reblogs
Allow hiding reblogs on a per-follow basis
2017-11-15 12:01:17 -05:00
David Yip
656d54e945 Maintain case-insensitivity when merging multiple matchers (#213)
When given two regexps, Regexp.union preserves the options set (or not
set) on each regex; this meant that none of the multiline (m),
case-insensitivity (i), or extended syntax (x) options were set.  Our
regexps are written expecting the m, i, and x options were set on all of
them, so we need to make sure that we preserve that behavior.
2017-11-13 11:06:02 -06:00
aschmitz
b95c48748c Per-user reblog hiding implementation/fixes/tests
Note that this will only hide/show *future* reblogs by a user, and does
nothing to remove/add reblogs that are already in the timeline. I don't
think that's a particularly confusing behavior, and it's a lot easier
to implement (similar to mutes, I believe).
2017-11-10 22:04:54 -06:00
David Yip
dd28b94cf0 Merge remote-tracking branch 'origin/master' into gs-master 2017-10-26 09:18:27 -05:00
unarist
4f337c020a Fix Cocaine::ExitStatusError when upload small non-animated GIF (#5489)
Looks like copied tempfile need to be flushed before further processing. This issue won't happen if the uploaded file has enough file size.
2017-10-26 22:48:35 +09:00
beatrix
a2612d0d38 Merge pull request #179 from glitch-soc/keyword-mute
Keyword muting
2017-10-25 17:37:48 -04:00
David Yip
42f2045c21 Merge remote-tracking branch 'STJrInuyasha/feature/direct-timeline' into gs-direct-timeline 2017-10-25 16:01:20 -05:00
David Yip
641f90e73a Fix example description.
This example actually checks matches at the end of a string.
2017-10-24 18:33:02 -05:00
Matthew Walsh
3db80f75a6 Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL
2017-10-22 18:35:14 -07:00
David Yip
4b68e82a19 Don't add \b to whole-word keywords that don't start with word characters.
Ditto for ending with \b.

Consider muting the phrase "(hot take)".  I stipulate it is reasonable
to enter this with the default "match whole word" behavior.  Under the
old behavior, this would be encoded as

    \b\(hot\ take\)\b

However, if \b is before the first character in the string and the first
character in the string is not a word character, then the match will
fail.  Ditto for after.  In our example, "(" is not a word character, so
this will not match statuses containing "(hot take)", and that's a very
surprising behavior.

To address this, we only add leading and trailing \b to keywords that
start or end with word characters.
2017-10-22 00:38:54 -05:00
David Yip
670e6a33f8 Move KeywordMute into Glitch namespace.
There are two motivations for this:

1. It looks like we're going to add other features that require
   server-side storage (e.g. user notes).

2. Namespacing glitchsoc modifications is a good idea anyway: even if we
   do not end up doing (1), if upstream introduces a keyword-mute feature
   that also uses a "KeywordMute" model, we can avoid some merge
   conflicts this way and work on the more interesting task of
   choosing which implementation to use.
2017-10-21 14:54:36 -05:00
David Yip
4a64181461 Allow keywords to match either substrings or whole words.
Word-boundary matching only works as intended in English and languages
that use similar word-breaking characters; it doesn't work so well in
(say) Japanese, Chinese, or Thai.  It's unacceptable to have a feature
that doesn't work as intended for some languages.  (Moreso especially
considering that it's likely that the largest contingent on the Mastodon
bit of the fediverse speaks Japanese.)

There are rules specified in Unicode TR29[1] for word-breaking across
all languages supported by Unicode, but the rules deliberately do not
cover all cases.  In fact, TR29 states

    For example, reliable detection of word boundaries in languages such
    as Thai, Lao, Chinese, or Japanese requires the use of dictionary
    lookup, analogous to English hyphenation.

So we aren't going to be able to make word detection work with regexes
within Mastodon (or glitchsoc).  However, for a first pass (even if it's
kind of punting) we can allow the user to choose whether they want word
or substring detection and warn about the limitations of this
implementation in, say, docs.

[1]: https://unicode.org/reports/tr29/
     https://web.archive.org/web/20171001005125/https://unicode.org/reports/tr29/
2017-10-21 14:54:36 -05:00
David Yip
9f609bc94e Fix case-insensitive match scenario; test some word ornamentation. #164. 2017-10-21 14:54:36 -05:00
David Yip
603cf02b70 Rework KeywordMute interface to use a matcher object; spec out matcher. #164.
A matcher object that builds a match from KeywordMute data and runs it
over text is, in my view, one of the easier ways to write examples for
this sort of thing.
2017-10-21 14:54:36 -05:00
David Yip
4745d6eeca Spec out KeywordMute interface. #164. 2017-10-21 14:54:21 -05:00
David Yip
9093e2de7a Add KeywordMute model.
Gist of the proposed keyword mute implementation:

Keyword mutes are represented server-side as one keyword per record.
For each account, there exists a keyword regex that is generated as one
big alternation of all keywords.  This regex is cached (in Redis, I
guess) so we can quickly get it when filtering in FeedManager.
2017-10-21 14:53:41 -05:00
David Yip
6cd5b3bbe5 Merge remote-tracking branch 'origin/master' into gs-master 2017-10-16 01:29:02 -05:00
Eugen Rochko
3283868e28 Improve spec of Feed and UserTrackingConcern (#5367) 2017-10-13 11:00:11 +02:00
kibigo!
8d6b9ba494 Merge upstream 2.0ish #165 2017-10-11 10:43:10 -07:00
nullkal
633426b261 Add moderation note (#5240)
* Add moderation note

* Add frozen_string_literal

* Make rspec pass
2017-10-07 20:26:43 +02:00
utam0k
b3af3f9f8c Implement EmailBlackList (#5109)
* Implement BlacklistedEmailDomain

* Use Faker::Internet.domain_name

* Remove note column

* Add frozen_string_literal comment

* Delete unnecessary codes

* Sort alphabetically

* Change of wording

* Rename BlacklistedEmailDomain to EmailDomainBlock
2017-10-04 15:16:10 +02:00
aschmitz
468523f4ad Non-Serial ("Snowflake") IDs (#4801)
* Use non-serial IDs

This change makes a number of nontrivial tweaks to the data model in
Mastodon:

* All IDs are now 8 byte integers (rather than mixed 4- and 8-byte)
* IDs are now assigned as:
  * Top 6 bytes: millisecond-resolution time from epoch
  * Bottom 2 bytes: serial (within the millisecond) sequence number
  * See /lib/tasks/db.rake's `define_timestamp_id` for details, but
    note that the purpose of these changes is to make it difficult to
    determine the number of objects in a table from the ID of any
    object.
* The Redis sorted set used for the feed will have values used to look
  up toots, rather than scores. This is almost always the same as the
  existing behavior, except in the case of boosted toots. This change
  was made because Redis stores scores as double-precision floats,
  which cannot store the new ID format exactly. Note that this doesn't
  cause problems with sorting/pagination, because ZREVRANGEBYSCORE
  sorts lexicographically when scores are tied. (This will still cause
  sorting issues when the ID gains a new significant digit, but that's
  extraordinarily uncommon.)

Note a couple of tradeoffs have been made in this commit:

* lib/tasks/db.rake is used to enforce many/most column constraints,
  because this commit seems likely to take a while to bring upstream.
  Enforcing a post-migrate hook is an easier way to maintain the code
  in the interim.
* Boosted toots will appear in the timeline as many times as they have
  been boosted. This is a tradeoff due to the way the feed is saved in
  Redis at the moment, but will be handled by a future commit.

This would effectively close Mastodon's #1059, as it is a
snowflake-like system of generating IDs. However, given how involved
the changes were simply within Mastodon, it may have unexpected
interactions with some clients, if they store IDs as doubles
(or as 4-byte integers). This was a problem that Twitter ran into with
their "snowflake" transition, particularly in JavaScript clients that
treated IDs as JS integers, rather than strings. It therefore would be
useful to test these changes at least in the web interface and popular
clients before pushing them to all users.

* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Restructure feed pushes/unpushes

This was necessary because the previous behavior used Redis zset scores
to identify statuses, but those are IEEE double-precision floats, so we
can't actually use them to identify all 64-bit IDs. However, it leaves
the code in a much better state for refactoring reblog handling /
coalescing.

Feed-management code has been consolidated in FeedManager, including:

* BatchedRemoveStatusService no longer directly manipulates feed zsets
* RemoveStatusService no longer directly manipulates feed zsets
* PrecomputeFeedService has moved its logic to FeedManager#populate_feed

(PrecomputeFeedService largely made lots of calls to FeedManager, but
didn't follow the normal adding-to-feed process.)

This has the effect of unifying all of the feed push/unpush logic in
FeedManager, making it much more tractable to update it in the future.

Due to some additional checks that must be made during, for example,
batch status removals, some Redis pipelining has been removed. It does
not appear that this should cause significantly increased load, but if
necessary, some optimizations are possible in batch cases. These were
omitted in the pursuit of simplicity, but a batch_push and batch_unpush
would be possible in the future.

Tests were added to verify that pushes happen under expected conditions,
and to verify reblog behavior (both on pushing and unpushing). In the
case of unpushing, this includes testing behavior that currently leads
to confusion such as Mastodon's #2817, but this codifies that the
behavior is currently expected.

* Rubocop fixes

I could swear I made these changes already, but I must have lost them
somewhere along the line.

* Address review comments

This addresses the first two comments from review of this feature:

https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336735
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336931

This adds an optional argument to FeedManager#key, the subtype of feed
key to generate. It also tests to ensure that FeedManager's settings are
such that reblogs won't be tracked forever.

* Hardcode IdToBigints migration columns

This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Only implement timestamp IDs for Status IDs

Per discussion in #4801, this is only being merged in for Status IDs at
this point. We do this in a migration, as there is no longer use for
a post-migration hook. We keep the initialization of the timestamp_id
function as a Rake task, as it is also needed after db:schema:load (as
db/schema.rb doesn't store Postgres functions).

* Change internal streaming payloads to stringified IDs as well

This is equivalent to 591a9af356faf2d5c7e66e3ec715502796c875cd from
#5019, with an extra change for the addition to FeedManager#unpush.

* Ensure we have a status_id_seq sequence

Apparently this is not a given when specifying a custom ID function,
so now we ensure it gets created. This uses the generic version of this
function to more easily support adding additional tables with timestamp
IDs in the future, although it would be possible to cut this down to a
less generic version if necessary. It is only run during db:schema:load
or the relevant migration, so the overhead is extraordinarily minimal.

* Transition reblogs to new Redis format

This provides a one-way migration to transition old Redis reblog entries
into the new format, with a separate tracking entry for reblogs.

It is not invertible because doing so could (if timestamp IDs are used)
require a database query for each status in each users' feed, which is
likely to be a significant toll on major instances.

* Address review comments from @akihikodaki

No functional changes.

* Additional review changes

* Heredoc cleanup

* Run db:schema:load hooks for test in development

This matches the behavior in Rails'
ActiveRecord::Tasks::DatabaseTasks.each_current_configuration, which
would otherwise break `rake db:setup` in development.

It also moves some functionality out to a library, which will be a good
place to put additional related functionality in the near future.
2017-10-04 09:56:37 +02:00
Eugen Rochko
887cd94e96 Increase attachment descriptions to 420 characters (#5139)
Blaze it
2017-09-29 02:30:00 +02:00
Eugen Rochko
4ec1771165 Add ability to specify alternative text for media attachments (#5123)
* Fix #117 - Add ability to specify alternative text for media attachments

- POST /api/v1/media accepts `description` straight away
- PUT /api/v1/media/:id to update `description` (only for unattached ones)
- Serialized as `name` of Document object in ActivityPub
- Uploads form adjusted for better performance and description input

* Add tests

* Change undo button blend mode to difference
2017-09-28 15:31:31 +02:00
Ondřej Hruška
83bda6c1a8
Merge commit 'b9f59ebcc68e9da0a7158741a1a2ef3564e1321e' into merging-upstream 2017-09-28 09:18:35 +02:00
Ondřej Hruška
9330ea1f4d
Merge commit '4aea3f88a6d30f102a79c2da7fcfac96465ba1a8' into merging-upstream 2017-09-28 09:12:17 +02:00
Akihiko Odaki
0de82dd316 Do not filter statuses with unknown languages (#5045) 2017-09-22 19:33:17 +02:00
Eugen Rochko
81cec35dbf Custom emoji (#4988)
* Custom emoji

- In OStatus: `<link rel="emoji" name="coolcat" href="http://..." />`
- In ActivityPub: `{ type: "Emoji", name: ":coolcat:", href: "http://..." }`
- In REST API: Status object includes `emojis` array (`shortcode`, `url`)
- Domain blocks with reject media stop emojis
- Emoji file up to 50KB
- Web UI handles custom emojis
- Static pages render custom emojis as `<img />` tags

Side effects:

- Undo #4500 optimization, as I needed to modify it to restore
  shortcode handling in emojify()
- Formatter#plaintext should now make sure stripped out line-breaks
  and paragraphs are replaced with newlines

* Fix emoji at the start not being converted
2017-09-19 02:42:40 +02:00
unarist
dd6f9a1b82 Validate uri presence for remote status (#4985) 2017-09-17 15:21:57 +02:00
Akihiko Odaki
198a9a4fa4 Remove local_only scope in Status (#4977) 2017-09-17 05:29:43 +02:00
Surinna Curtis
d880b3182b minor code style fixes oops 2017-09-13 21:47:30 -05:00
Surinna Curtis
0f2af2a974 Add specs for how mute! interacts with muting_notifications? 2017-09-13 21:47:30 -05:00
Eugen Rochko
9239e4ce4d Uploads for admin site settings (#4913)
* Improve OpenGraph tags for about pages

* Add thumbnail admin setting

* Fix error

* Fix up
2017-09-14 00:04:30 +02:00
David Yip
67ad453373 Merge tag 'v1.6.0rc4' into sync/upstream-1.6.0rc4
Conflicts:
      app/javascript/mastodon/features/getting_started/index.js
      app/javascript/packs/public.js
      app/javascript/styles/components.scss
2017-09-09 23:56:21 -05:00
David Yip
514fc908a3 Merge tag 'v1.6.0rc3' into sync/upstream 2017-09-09 14:28:08 -05:00
David Yip
b9f7bc149b Merge branch 'origin/master' into sync/upstream
Conflicts:
	app/javascript/mastodon/components/status_list.js
	app/javascript/mastodon/features/notifications/index.js
	app/javascript/mastodon/features/ui/components/modal_root.js
	app/javascript/mastodon/features/ui/components/onboarding_modal.js
	app/javascript/mastodon/features/ui/index.js
	app/javascript/styles/about.scss
	app/javascript/styles/accounts.scss
	app/javascript/styles/components.scss
	app/presenters/instance_presenter.rb
	app/services/post_status_service.rb
	app/services/reblog_service.rb
	app/views/about/more.html.haml
	app/views/about/show.html.haml
	app/views/accounts/_header.html.haml
	config/webpack/loaders/babel.js
	spec/controllers/api/v1/accounts/credentials_controller_spec.rb
2017-09-09 14:27:47 -05:00
voidSatisfaction
8185f98872 Feat add validation for report comment: characters under 1000 valid (#4833) 2017-09-07 09:55:42 +02:00
Eugen Rochko
e7adbf572a Switch to static URIs, new URI format in both protocols for new statuses (#4815)
* Decouple Status#local? from uri being nil

* Replace on-the-fly URI generation with stored URIs

- Generate URI in after_save hook for local statuses
- Use static value in TagManager when available, fallback to tag format
- Make TagManager use ActivityPub::TagManager to understand new format
- Adjust tests

* Use other heuristic for locality of old statuses, do not perform long query

* Exclude tombstone stream entries from Atom feed

* Prevent nil statuses from landing in Pubsubhubbub::DistributionWorker

* Fix URI not being saved (#4818)

* Add more specs for Status

* Save generated uri immediately

and also fix method order to minimize diff.

* Fix alternate HTML URL in Atom

* Fix tests

* Remove not-null constraint from statuses migration to speed it up
2017-09-06 19:01:28 +02:00
abcang
6ec1aa372d Validate data of Imports (#4782) 2017-09-02 20:45:42 +02:00
Eugen Rochko
9caa90025f Pinned statuses (#4675)
* Pinned statuses

* yarn manage:translations
2017-08-25 01:41:18 +02:00
Colin Mitchell
871c0d251a Application prefs section (#2758)
* Add code for creating/managing apps to settings section

* Add specs for app changes

* Fix controller spec

* Fix view file I pasted over by mistake

* Add locale strings. Add 'my apps' to nav

* Add Client ID/Secret to App page. Add some visual separation

* Fix rubocop warnings

* Fix embarrassing typo

I lost an `end` statement while fixing a merge conflict.

* Add code for creating/managing apps to settings section

- Add specs for app changes
- Add locale strings. Add 'my apps' to nav
- Add Client ID/Secret to App page. Add some visual separation
- Fix some bugs/warnings

* Update to match code standards

* Trigger notification

* Add warning about not sharing API secrets

* Tweak spec a bit

* Cleanup fixture creation by using let!

* Remove unused key

* Add foreign key for application<->user
2017-08-22 18:33:57 +02:00
Eugen Rochko
3534e115e5 Do not try to re-subscribe to unsubscribed accounts (#4653) 2017-08-21 17:32:41 +02:00
kibigo!
8150689b48 Merge upstream (#111) 2017-08-01 13:20:29 -07:00
Eugen Rochko
53b2b1b238 Count all URLs in text as 23 characters flat, do not count domain part of usernames (#4427)
* Count all URLs in text as 23 characters flat, do not count domain part of usernames

* Add new status text counting logic to web UI
2017-07-29 00:06:29 +02:00
Ondřej Hruška
d69fa9e1f4 Merge changes from upstream with the CSS reload fix 2017-07-18 18:58:47 +02:00
Yamagishi Kazutoshi
3267e4a785 Add unfollow modal (optional) (#4246)
* Add unfollow modal

* unfollowing someone

* remove unnecessary prop
2017-07-18 17:14:43 +02:00
abcang
4d42a38954 Improve admin page (#4121)
* Improve admin page

* Fix test

* Add spec

* Improve select style
2017-07-18 16:38:22 +02:00