Commit graph

350 commits

Author SHA1 Message Date
dullbananas c4fc3a8ede
Optimize stuff in attempt to fix high amount of locks, and fix comment_aggregates.child_count (#4696)
* separate triggers

* auto_explain.log_triggers=on

* Revert "auto_explain.log_triggers=on"

This reverts commit 078b2dbb9b.

* Revert "separate triggers"

This reverts commit 95600da4af.

* bring back migration

* re-order statements

* add comment about statement ordering

* no redundant updates

* optimize post_aggregates update in comment trigger

* set comment path in trigger

* update comment_aggregates.child_count using trigger

* move `LEFT JOIN post` to inner query

* clean up newest_comment_time_necro

* add down.sql
2024-05-09 08:18:55 -04:00
SleeplessOne1917 b152be7951
Update rustls (#4690)
* Update rustls

* Format code
2024-05-03 16:06:14 -04:00
Nutomic beec080274
Testing for federation with NodeBB, make community.followers_url optional (#4629)
* Testing for federation with NodeBB, make community.followers_url optional

* clippy
2024-04-29 12:34:11 +02:00
Dessalines 492d8f1b01
Fix communities with broken outboxes, and use PostView. Fixes #4658 (#4668)
* Fix communities with broken outboxes, and use PostView. Fixes #4658

* Fixing tests.

* Dont pass ref and clone.
2024-04-29 12:22:00 +02:00
dullbananas d3737d4453
Optimize actor_language.rs (#4612)
* Remove useless transaction in actor_language.rs

* Update actor_language.rs

* site

* community

* Update actor_language.rs

* Update actor_language.rs

* Update actor_language.rs

* Update actor_language.rs

* Update actor_language.rs
2024-04-27 10:59:58 -04:00
dullbananas 4ba6221e04
Move SQL triggers from migrations into reusable sql file (#4333)
* stuff

* stuff including batch_upsert function

* stuff

* do things

* stuff

* different timestamps

* stuff

* Revert changes to comment.rs

* Update comment.rs

* Update comment.rs

* Update post_view.rs

* Update utils.rs

* Update up.sql

* Update up.sql

* Update down.sql

* Update up.sql

* Update main.rs

* use anyhow macro

* Create down.sql

* Create up.sql

* Create replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update utils.rs

* Update .woodpecker.yml

* Update sql_format_check.sh

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Create dump_schema.sh

* Update start_dev_db.sh

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* stuff

* Update replaceable_schema.sql

* Update .pg_format

* fmt

* stuff

* stuff (#21)

* Update replaceable_schema.sql

* Update up.sql

* Update replaceable_schema.sql

* fmt

* update cargo.lock

* stuff

* Update replaceable_schema.sql

* Remove truncate trigger because truncate is already restricted by foreign keys

* Update replaceable_schema.sql

* fix some things

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update .woodpecker.yml

* stuff

* fix TG_OP

* Psql env vars

* try to fix combine_transition_tables parse error

* Revert "try to fix combine_transition_tables parse error"

This reverts commit 75d00a4626.

* refactor combine_transition_tables

* try to fix create_triggers

* fix some things

* try to fix combined_transition_tables

* fix sql errors

* update comment count in post trigger

* fmt

* Revert "fmt"

This reverts commit a5bcd0834b.

* Revert "update comment count in post trigger"

This reverts commit 0066a4b42b.

* fix everything

* Update replaceable_schema.sql

* actually fix everything

* refactor create_triggers

* fix

* add semicolons

* add is_counted function and fix incorrect bool operator in update_comment_count_from_post

* refactor comment trigger

* refactor post trigger

* fix

* Delete crates/db_schema/src/utils/series.rs

* subscribers_local

* edit migrations

* move migrations

* remove utils::series module declaration

* fix everything

* stuff

* Move sql to schema_setup dir

* utils.sql

* delete .pg_format

* Update .woodpecker.yml

* Update sql_format_check.sh

* Update .woodpecker.yml

* Merge remote-tracking branch 'upstream/main' into bliss

* fmt

* Create main.rs

* Update lib.rs

* Update main.rs

* Update .woodpecker.yml

* Update main.rs

* Update Cargo.toml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update triggers.sql

* YAY

* Update mod.rs

* Update Cargo.toml

* a

* Update Cargo.toml

* Update Cargo.toml

* Delete crates/db_schema/src/main.rs

* Update Cargo.toml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update utils.sql

* Update utils.sql

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update down.sql

* Update up.sql

* Update triggers.sql

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update triggers.sql

* Update down.sql

* Update .woodpecker.yml

* Update Cargo.toml

* Update .woodpecker.yml

* Update Cargo.toml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update mod.rs

* Update Cargo.toml

* Update mod.rs

* make dump_schema.sh executable

* fix dump_schema.sh

* defer

* diff dumps

* fmt

* Update utils.sql

* Update .woodpecker.yml

* use correct version for pg_dump

* Update .woodpecker.yml

* Update .woodpecker.yml

* change migration date

* atomic site_aggregates insert

* temporarily repeat tests in CI

* drop r schema in CI migration check

* show ReceivedActivity::create error

* move check_diesel_migration CI step

* Update .woodpecker.yml

* Update scheduled_tasks.rs

* Update scheduled_tasks.rs

* update cargo.lock

* move sql files

* move rank functions

* filter post_aggregates update

* fmt

* cargo fmt

* replace post_id with id

* update cargo.lock

* avoid locking rows that need no change in up.sql

* only run replaceable_schema if migrations were run

* debug ci test failure

* make replaceable_schema work in CI

* Update .woodpecker.yml

* remove println

* Use migration revert and git checkout

* Update schema_setup.rs

* Fix

* Update schema_setup.rs

* Update schema_setup.rs

* Update .woodpecker.yml

---------

Co-authored-by: Nutomic <me@nutomic.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-04-17 20:58:44 -04:00
TechVest b0370ae2fd
chore: fix some comments (#4637)
Signed-off-by: TechVest <techdashen@qq.com>
2024-04-17 14:35:54 +02:00
Dessalines 6efab9aab1
Creating a LocalImageView, so that front ends have the Person struct. (#4631)
* Creating a LocalImageView, so that front ends have the Person struct.

* Removing local_user from LocalImageView.

* Add uploader check.
2024-04-16 19:20:44 -04:00
Dessalines d075acce43
Make all single-fetch database calls return an Option. (#4617)
- Diesel ordinarily throws an error when no results are returned for a
  single fetch, which is a bit confusing. This PR ensures that the
  missing value cases are all caught, and wrapped with new LemmyErrors,
  rather than diesel errors.
- Fixes #4601
2024-04-16 14:48:15 +02:00
Dessalines 5dea21d531
Convert all Result<..., LemmyError> into LemmyResult<...> Fixes #4613 (#4614)
* Convert all Result<..., LemmyError> into LemmyResult<...> Fixes #4613

* Fixing clippy.
2024-04-10 10:14:11 -04:00
Nutomic 0203b62a6d
Ignore old federated post edits (ref #4529) (#4586)
* Ignore old federated post edits (ref #4529)

* use filter on insert

* coalesce(updated, published)

* avoid comment conflict clause

---------

Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
2024-04-10 10:03:51 -04:00
Nutomic 1d0a6ac08f
Avoid breaking api change, reduce api cache duration (#4610)
* Dont mark site.public_key as `serde(skip)` to avoid breaking change (fixes #4605)

* Reduce cache duration for api
2024-04-09 10:10:20 -04:00
Nutomic 087684658a
Cache result of LocalSite::read to avoid unnecessary db calls (#4585)
* Cache result of LocalSite::read to avoid unnecessary db calls

* single const for cache duration

* clippy

* revert apub send changes

* clippy

* fmt
2024-04-03 17:38:31 -04:00
dullbananas 007e9b7aab
Optimize Community::set_featured_posts (#4579)
* Don't lock excess rows in Community::set_featured_posts

* Update community.rs

* Update community.rs

* Update community.rs

* Update community.rs
2024-04-02 11:19:04 -04:00
Nutomic 85ee89f4e8
When uploading new icon/avatar/banner, delete old one (#4573) 2024-03-27 09:00:52 -04:00
Dessalines 6bfbb9332d
Adding listMedia endpoint, to view all your local image uploads. (#4509)
* Adding listMedia endpoint, to view all your local image uploads.

- Fixes #4445

* Fix ts import.

* Forgot to order by published desc

* Adding an endpoint to list all images, for admins only.

* Forgot to add file.

* Add additional test.

* Use better logic for no-limit version.

* Better call sites.

* Adding another test.

* Fix tests.

* Moving list_media to /account action.

* Addressing PR comments.

* Removing pointless comment.

---------

Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
2024-03-26 12:06:11 -04:00
Dessalines 7929e77602
Fixing issue with comment replies wrongly marked as read. (#4567)
* Fixing issue with comment replies wrongly marked as read.

- Fixes #4566

* Elaborating on a comment.
2024-03-26 10:46:37 -04:00
Dessalines 95069d7648
Fixing some clippy and woodpecker lints. (#4565)
* Fixing some clippy and woodpecker lints.

* Try fixing woodpecker 1.

* Revert "Try fixing woodpecker 1."

This reverts commit 7c2020a08d.
2024-03-26 10:17:42 +01:00
Nutomic 846848c4f6
On registration, automatically set content languages from accept-language header (#4550)
* On registration, automatically set content languages from accept header

* no need to set site language or default language for new user anymore

* fix test

* fix langs

* avoid duplicate writing of new user languages
2024-03-25 16:02:12 -04:00
Dessalines 0f77951e05
Upgrading deps. (#4537)
* Upgrading deps.

* Addressing PR comments
2024-03-18 10:36:49 +01:00
flamingos-cant 19a1a077c5
Add a blocklist for URLs. (#4515)
* Add a blocklist for URLs.

* Fix SQL format

* Make clippy happy.

* Use regex for URL matching.

* Escape regex chars in URLs.

* Use post for modification.

* Make URL block regex static and remove API routes.

* Add date fields to table and use transaction.

* Use Cache for blocklist.

* Rename check_links + move list to parameters of process_markdown.

* SQL format.

* Format, again.

* Remove println.

* Add API test.

* Set a shorter lifetime for regex in debug mode.

* Add missing macro.

* Update lemmy-js-client

* Update api_test/pnpm-lock.yaml

* Don't break other tests

* Use different URL for test

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Nutomic <me@nutomic.com>
2024-03-15 07:03:29 -04:00
Dessalines 15f02f00a9
Add a vote_display_mode local_user setting. (#4450)
* Add a vote_display_mode local_user setting.

- Fixes #4449

* Changing HideDownvotes to Score.

* Adding ScoreAndDownvote display mode.

* Adding upvote and downvote mode.

* Extracting vote_display_mode to another table.

* Fixing fmt.

* Remove published and updated columns.

---------

Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
2024-03-13 12:10:58 -04:00
Nutomic 5859502a2a
Fix missing private key for signed fetch (#4516)
* Fix missing private key for signed fetch (fixes #4451)

* clippy

* instance actor name and webfinger

* better webfinger handling

* upgrade lib

* update test asset
2024-03-08 10:23:15 -05:00
Nutomic 00f7778485
Store thumbnails in db table local_image (#4512)
* Store thumbnails in db table local_image

* fmt
2024-03-08 10:17:26 -05:00
battmdpkq f228f9d7a9
fix some typos (#4519)
Signed-off-by: battmdpkq <cmaker@163.com>
2024-03-08 10:38:20 +01:00
Dessalines 7f9950fe85
Add alt_text for posts. Fixes #1086 (#4477)
* Add alt_text for posts. Fixes #1086

* Moving alt_text to attachment name.

* Cleaning up mod action line.

* Addressing PR comments

* Addressing PR comments.

* Fixing clones.
2024-03-05 11:34:57 +01:00
SleeplessOne1917 36ad1868b3
Make SiteAggregates derive Copy and Hash (#4501)
Co-authored-by: SleeplessOne1917 <insomnia-void@protonmail.com>
2024-03-05 11:14:12 +01:00
Dessalines 7eec8714d7
When site banning a federated user, also remove their content from our local communities. (#4464)
* When banning a federated user, also remove their content from our local
communities.

- This works by:
  - Before a site ban, find all posts and comments to local communities
  - Send a federated community ban action for each local comm.
  - This also removes their content in the apub receive code.
- Adding back in federated community ban api tests.
- Adding in two more api tests for site bans.
- Fixes #4118

* Add local community ban, and nonlocal person check.

* Ignoring errors.

* Move local check into function.

* Addressing PR comments 2
2024-03-01 13:45:06 -05:00
Dessalines 87b577467b
Adding ability to hide posts. (#4480)
* Adding ability to hide posts.

- Adds an post/hide API route.
- Adds a `show_hidden` (default false) to `GetPosts`.
- Adds a `hidden` field to `PostView`.
- Removes the single `post_id` from MarkPostAsRead.
- Fixes #1403

* Add a check to make sure hidden field is true.

* Fixing test.

* Add back semicolon
2024-02-29 10:42:34 -05:00
Nutomic 328a48c9f5
Remove error-type feature from lemmy-utils (#4474)
* Remove error-type feature from lemmy-utils

* fixes

---------

Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
2024-02-27 11:13:52 -05:00
Dessalines f3d48f2c2c
Adding some recommended fixes from nightly clippy. (#4473) 2024-02-26 09:47:10 -05:00
Nutomic 7316dd281a
Correctly synchronize collection of community featured posts (fixes #3867) (#4475) 2024-02-26 09:45:23 -05:00
SleeplessOne1917 f42420809b
Expose LemmyErrorType in lemmy_api_common (#4439)
* Expose LemmyErrorType in lemmy_api_common

* Make conditional compilation gates for utils

* Make it so api_common doesn't pull in unnecessary deps

* Make error type non exhaustive

* Fix formatting

* Format toml

* Add some convenience derives to LemmyError

* Simplify features

* Fix CI compile error

---------

Co-authored-by: SleeplessOne1917 <insomnia-void@protonmail.com>
2024-02-24 19:54:27 -05:00
dullbananas f56b84615c
Move DbUrl trait impls to newtypes.rs (#4463)
* Move DbUrl trait impls to newtypes.rs

* Update utils.rs
2024-02-19 12:41:28 -05:00
dullbananas d79502dff3
Escape backslashes in fuzzy_search (#4462)
* Escape backslashes in fuzzy_search

* Update utils.rs
2024-02-18 09:12:56 -05:00
Dessalines 5d551e6da5
Adding an instance-level default_sort_type (#4454)
* Adding an instance-level default_sort_type

- Fixes #3796

* Fixing comment.

* Put user sort before site sort.
2024-02-16 09:36:46 -05:00
Nutomic ffcf415cac
Dont log db url on connection error (fixes #4453) (#4456)
* Dont log db url on connection error (fixes #4453)

* remove format
2024-02-16 08:50:06 -05:00
Nutomic 86b44c2a4d
Add site.content_warning, local_site.default_post_listing_mode (#4393)
* Include local_site.content_warning setting for showing nsfw by default

* Add community setting `only_followers_can_vote`

* clippy

* add auto_expand_images site setting

* cleanup

* add missing api params

* postquery/communityquery changes

* clippy

* change error

* replace auto_expand_images with default_site_post_listing_mode

* change post/community query params

* get rid of only_followers_can_vote

* machete

* fix

* clippy

* revert remaining vote changes

* remove dead code

* remove unused var

* fmt
2024-02-16 07:24:35 -05:00
Dessalines a3bf2f1cf1
Auto resolve reports on removing a comment or post. Fixes #4390 (#4402)
* Automatically resolve report when post/comment is removed (#3850)

* Automatically resolve report when post/comment is removed

* also handle apub removes

* Removing auto-resolve report triggers.

* Dont allow creating reports for deleted / removed items.

* Running pgformat.

* Fixing test.

* Addressing PR comments.

* Forgot comment report.

---------

Co-authored-by: Nutomic <me@nutomic.com>
2024-02-15 08:52:04 -05:00
Dessalines 890565ca14
Deleting denied local_users older than a week. Fixes #4434 (#4448)
* Deleting denied local_users older than a week. Fixes #4434

* Addressing PR comments.

* Upping rust to 1.76

* Delete the person rows also.
2024-02-15 07:50:53 -05:00
Dessalines 33989f5518
Blocking an instance also hides private messages from their users. (#4447)
* Blocking an instance also hides private messages from their users.

- Fixes #4444

* Separating private message tests.
2024-02-14 10:49:55 +01:00
dullbananas 677d54ae57
Allow better query plans (#4424)
* Update utils.rs

* Create bind_if_some.rs

* limit connection age

* Delete crates/db_schema/src/utils/bind_if_some.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs
2024-02-12 16:44:29 +01:00
SleeplessOne1917 300869d397
Make request models derive PartialEq, Eq, and Hash (#4443)
* Make request models derive PartialEq, Eq, and Hash

* Fix clippy error

---------

Co-authored-by: SleeplessOne1917 <insomnia-void@protonmail.com>
2024-02-11 00:32:14 -05:00
Nutomic 0f414a95d5
Local only community (#4350)
* Add support for local only community (fixes #1576)

* add filters and tests to db views

* dont federate local only community

* test get apub community http

* tests

* more checks

* wip

* api test

* fix tests

* change community.local_only column to visibility enum
(for private communities)

* sql fmt

* rename vars

* clippy

* fix tests

* update lib

* review

* fix js client version

* update client
2024-01-25 11:04:25 -05:00
Nutomic e8a52d3a5c
Rewrite images to use local proxy (#4035)
* Add markdown rule to add rel=nofollow for all links

* Add markdown image rule to add local image proxy (fixes #1036)

* comments

* rewrite markdown image links working

* add comment

* perform markdown image processing in api/apub receivers

* clippy

* add db table to validate proxied links

* rewrite link fields for avatar, banner etc

* sql fmt

* proxy links received over federation

* add config option

* undo post.url rewriting, move http route definition

* add tests

* proxy images through pictrs

* testing

* cleanup request.rs file

* more cleanup (fixes #2611)

* include url content type when sending post over apub (fixes #2611)

* store post url content type in db

* should be media_type

* get rid of cache_remote_thumbnails setting, instead automatically
take thumbnail from federation data if available.

* fix tests

* add setting disable_external_link_previews

* federate post url as image depending on mime type

* change setting again

* machete

* invert

* support custom emoji

* clippy

* update defaults

* add image proxy test, fix test

* fix test

* clippy

* revert accidental changes

* address review

* clippy

* Markdown link rule-dess (#4356)

* Extracting opengraph_data to its own type.

* A few additions for markdown-link-rule.

---------

Co-authored-by: Nutomic <me@nutomic.com>

* fix setting

* use enum for image proxy setting

* fix test configs

* add config backwards compat

* clippy

* machete

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-25 09:22:11 -05:00
dullbananas d8f9e8a64c
Post view: move cursor pagination to separate library, add backward pagination to PostQuery (#4320)
* stuff

* stuff

* crates.io

* Update up.sql

* Rerun federation tests

* Update post_view.rs

* Update post_view.rs

* Update up.sql

* Update utils.rs

* Fix precision loss

* Update up.sql

* Update down.sql

* remove unwrap

* Update post_view.rs

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-24 10:50:11 -05:00
dullbananas 759f6d8a9a
Better query plan viewing experience (#4285)
* stuff

* stuff including batch_upsert function

* stuff

* do things

* stuff

* different timestamps

* stuff

* Revert changes to comment.rs

* Update comment.rs

* Update comment.rs

* Update post_view.rs

* Update utils.rs

* Update up.sql

* Update up.sql

* Update down.sql

* Update up.sql

* Update main.rs

* use anyhow macro

* replace get(0) with first()

* as_slice

* Update series.rs

* Update db_perf.sh

* Update and rename crates/db_schema/src/utils/series.rs to crates/db_perf/src/series.rs

* Update utils.rs

* Update main.rs

* Update main.rs

* Update .woodpecker.yml

* fmt main.rs

* Update .woodpecker.yml

* Instance::delete at end

* Update main.rs

* Update Cargo.toml

---------

Co-authored-by: Nutomic <me@nutomic.com>
2024-01-24 10:22:33 -05:00
İsmail Karslı 8670403a67
Add local_subscribers field to CommunityAggregates. Fixes #4144 (#4166)
* Add upload timeout to PictrsConfig

* Bad space 🤔

* Update PictrsConfig upload timeout to include units.

* Add local_subscribers field to CommunityAggregates
struct and schema

* sql format

* local_subscribers test

* fix local_subscribers test

* Revert "fix local_subscribers test"

This reverts commit 4bbac5ce4a.

* Revert "local_subscribers test"

This reverts commit 735107e1f7.

* Create trigger for local_subscribers

* Rename variable

* re-trigger ci

* re-trigger ci

* Add local_subscribers count to follow.spec.ts

* Rename local_subscribers to subscribers_local

* Add subscribers_local to community_aggregates

* added subscribers_local to the aggregate tests

* Check if person exists on community_follower trigger

* Delete community follows before deleting person

* Update lemmy-js-client in api_tests

* Refactor local_subscriber migration

* fix format

* Move migration files date to now

* Fix test to wait for aggregates to federate

* re-trigger ci

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-24 10:22:05 -05:00
Nutomic b58da11fb7
Mark instance as alive after successful activity send (fixes #4039) (#4377)
* Mark instance as alive after successful activity send (fixes #4039)

* clippy

* Instance::update

* domain

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-19 09:40:12 -05:00
Dessalines 3b717cfc88
Removing serde_skip from newest_comment_time. Fixes #4351 (#4360) 2024-01-10 10:44:27 +01:00