bonfire-app/docs/HACKING.md

283 lines
14 KiB
Markdown
Raw Normal View History

2021-05-22 14:36:40 +00:00
# Development guide
_These instructions are for hacking on Bonfire. If you wish to deploy in production, please refer to our [Deployment Guide](./DEPLOY.md)!_
Hello, potential contributor! :-)
2022-01-09 10:19:16 +00:00
This is a work in progress guide to getting up and running as a
developer. Please ask questions in the issue tracker if something is
not clear and we'll try to improve it.
2021-05-22 14:36:40 +00:00
Happy hacking!
2022-01-09 10:19:16 +00:00
## Status: alpha - have fun but don't run in production.
2021-05-22 14:36:40 +00:00
2022-01-09 10:19:16 +00:00
Bonfire is currently alpha software. While it's fun to play with it,
we would *absolutely not* recommend running any production instances
yet because it's just not ready for that today.
2021-05-22 14:36:40 +00:00
2022-02-24 18:49:49 +00:00
The team is working hard on a beta release of bonfire social (classic flavour)
which should be ready in Q1 2022. Other flavours/apps will probably lag
2022-01-09 10:19:16 +00:00
behind for a while and may not always build cleanly. We recommend
sticking with bonfire social (aka the 'classic' flavour) for now.
2021-05-22 14:36:40 +00:00
2022-02-24 18:49:49 +00:00
## Configuration
2021-05-22 14:36:40 +00:00
2022-02-24 18:49:49 +00:00
### Picking a flavour
2021-05-22 16:18:42 +00:00
2022-02-24 18:49:49 +00:00
Bonfire is a flexible platform that powers a variety of social networks. The first thing you have to choose is which app (or "flavour") you want to hack on:
2021-05-22 16:18:42 +00:00
2022-02-24 18:49:49 +00:00
For example if you want to run the `classic` flavour:
2021-05-22 16:18:42 +00:00
2022-02-24 18:49:49 +00:00
`export FLAVOUR=classic`
2022-01-09 10:19:16 +00:00
2022-02-24 18:49:49 +00:00
### Initialise
2022-01-09 10:19:16 +00:00
2022-02-24 18:49:49 +00:00
Yu need to clone this repository and change into the directory and then do some configuration:
2022-01-09 10:19:16 +00:00
2022-02-24 18:49:49 +00:00
```sh
$ git clone https://github.com/bonfire-networks/bonfire-app.git bonfire
$ cd bonfire
```
2022-01-09 10:19:16 +00:00
2022-02-24 18:49:49 +00:00
- Then run this command to initialise some default config (.env files which won't be checked into git):
2021-05-22 14:36:40 +00:00
`make pre-config`
2021-05-22 16:18:42 +00:00
- Edit the config (especially the secrets) for the current flavour in these files:
2021-05-22 14:36:40 +00:00
- `config/dev/secrets.env`
- `config/dev/public.env`
### Option A - the entry way (fully managed via docker-compose, recommended when you're first exploring)
- Dependencies:
2021-07-03 08:04:49 +00:00
2021-05-22 14:36:40 +00:00
- `make`
2021-11-13 07:42:51 +00:00
- Recent versions of Docker & [docker-compose](https://docs.docker.com/compose/install/)
2021-05-22 14:36:40 +00:00
- Make sure you've edited your .env files (see above) before getting started and proceed to Hello world!
### Option B - the easy way (with docker-managed database & search index, recommended for active development)
- Dependencies:
2021-07-03 08:04:49 +00:00
- `make`
2021-11-13 07:42:51 +00:00
- Recent versions of [Elixir](https://elixir-lang.org/install.html) (1.12+) and OTP/erlang (24+)
- Recent versions of [Rust](https://www.rust-lang.org/tools/install) and Cargo
- [pnpm](https://pnpm.io)
- Recent versions of Docker & [docker-compose](https://docs.docker.com/compose/install/)
2021-05-22 14:36:40 +00:00
- Set an environment variable to indicate your choice: `export WITH_DOCKER=partial`
- Make sure you've edited your .env files (see above) before getting started and proceed to Hello world!
### Option C - the bare metal (if you don't use docker)
- Dependencies:
2021-07-03 08:04:49 +00:00
- `make`
2021-11-13 07:42:51 +00:00
- Recent versions of [Elixir](https://elixir-lang.org/install.html) (1.12+) and OTP/erlang (24+)
- Recent versions of [Rust](https://www.rust-lang.org/tools/install) and Cargo
- [pnpm](https://pnpm.io)
- Postgres 12+ (or rather [Postgis](https://postgis.net/install/) if using the bonfire_geolocate extension)
- [Meili Search](https://docs.meilisearch.com/learn/getting_started/installation.html) (optional)
2021-05-22 14:36:40 +00:00
- You will need to set the relevant environment variables in the .env files (see above) to match your local install of Postgres.
- If you want search capabilities, you'll also need to setup a Meili server and set the relevant env variables as well.
- Set an environment variable to indicate your choice: `export WITH_DOCKER=no` and proceed to Hello world!
2021-05-22 16:06:31 +00:00
### Option D - the experimental one (dev environment with Nix)
2021-05-22 16:01:58 +00:00
2021-07-04 06:00:22 +00:00
- run a recent version of Nix or NixOS: https://nixos.wiki
- enable Flakes: https://nixos.wiki/wiki/Flakes#Installing_flakes
- add `sandbox = false` in your nix.conf
2021-07-03 08:04:49 +00:00
If you use direnv, just cd in the directory and you will have all the dependencies.
If you just have nix, running `nix shell .` (inside the repository) will set you up with a shell.
2021-05-22 16:01:58 +00:00
2021-07-04 06:00:22 +00:00
You will need to create and init the db directory (keeping all your Postgres data inside this directory):
2021-05-22 16:01:58 +00:00
- create the db directory `initdb ./db`
- create the postgres user `createuser postgres -ds`
- create the db `createdb bonfire_dev`
- start the postgres instance `pg_ctl -l "$PGDATA/server.log" start`
- `mix deps.get` to get elixir dependencies
- `pushd assets && npm install && popd` to get the frontend dependencies
- `mix ecto.migrate` to compile & get an up to date database
- `iex -S mix phx.server` to start the server
- check out the app on `localhost:4000` in your browser
2021-05-22 14:36:40 +00:00
## Hello world!
- From a fresh checkout of this repository, this command will fetch the app's dependencies and setup the database (the same commands apply for all three options above):
```
make setup
```
- You should then be able to run the app with:
```
make dev
```
2021-05-22 16:01:58 +00:00
- See the `make` commands below for more things you may want to do.
2021-05-22 14:36:40 +00:00
## Onboarding
By default, the back-end listens on port 4000 (TCP), so you can access it on http://localhost:4000/
If you haven't set up transactional emails, while in development, you will see a signup confirmation path appear in the iex console.
Note that the first account to be registered is automatically an instance admin.
## Documentation
The code is somewhat documented inline. You can generate HTML docs (using `Exdoc`) by running `mix docs`.
2021-05-22 16:01:58 +00:00
## Additional information
2021-07-03 08:04:49 +00:00
- messctl is a little utility for programmatically updating the .deps files from which the final elixir dependencies list is compiled by the mess script. The only use of it is in the dep-\* tasks of the Makefile. It is used by some of the project developers and the build does not rely on it.
2021-05-22 16:01:58 +00:00
2021-07-03 08:04:49 +00:00
- `./forks/` is used to hack on local copies of dependencies. You can clone a dependency from its git repo (like a bonfire extension) and use the local version during development, eg: `make dep.clone.local dep=bonfire_me repo=https://github.com/bonfire-networks/bonfire_me`
2021-05-22 16:01:58 +00:00
2021-12-19 05:51:11 +00:00
- You can migrate the DB when the app is running (useful in a release): `EctoSparkles.ReleaseTasks.migrate`
2021-05-22 16:01:58 +00:00
### Usage under Windows (MSYS or CYGWIN)
2021-07-03 08:04:49 +00:00
If you plan on using the `Makefile` (its rather handy), you must have symlinks enabled.
2021-05-22 16:01:58 +00:00
You must enable developer mode, and set `core.symlink = true`, [see link.](https://stackoverflow.com/a/59761201)
2021-05-22 14:36:40 +00:00
## Make commands
Run `make` followed by any of these commands when appropriate rather than directly using the equivalent commands like `mix`, `docker`, `docker-compose`, etc. For example, `make setup` will get you started, and `make dev` will run the app.
2021-05-22 14:42:44 +00:00
You can first set an env variable to control which mode these commands will assume you're using. Here are your options:
2021-07-03 08:04:49 +00:00
- `WITH_DOCKER=total` : use docker for everything (default)
- `WITH_DOCKER=partial` : use docker for services like the DB
- `WITH_DOCKER=easy` : use docker for services like the DB & compiled utilities like messctl
- `WITH_DOCKER=no` : please no
2021-05-22 14:36:40 +00:00
2021-05-22 14:41:05 +00:00
```
2021-05-22 14:36:40 +00:00
make help Makefile commands help **(run this to get more up-to-date commands and help information than available in this document)**
2021-05-22 16:06:31 +00:00
make mix~help Help info for elixir's mix commands
2021-05-22 14:36:40 +00:00
make env.exports Display the vars from dotenv files that you need to load in your environment
2021-07-03 08:04:49 +00:00
2021-05-22 14:36:40 +00:00
make setup First run - prepare environment and dependencies
make dev Run the app in development
make dev.bg Run the app in dev mode, as a background service
make db.reset Reset the DB (caution: this means DATA LOSS)
make db.rollback Rollback previous DB migration (caution: this means DATA LOSS)
make db.rollback.all Rollback ALL DB migrations (caution: this means DATA LOSS)
make update Update the app and all dependencies/extensions/forks, and run migrations
2021-07-03 08:04:49 +00:00
2021-05-22 14:36:40 +00:00
make update.app Update the app and Bonfire extensions in ./deps
2021-07-03 08:04:49 +00:00
make update.deps.bonfire Update to the latest Bonfire extensions in ./deps
2021-05-22 14:36:40 +00:00
make update.deps.all Update evey single dependency (use with caution)
make update.dep~% Update a specify dep (eg. `make update.dep~pointers`)
make update.forks Pull the latest commits from all ./forks
2021-07-03 08:04:49 +00:00
2021-05-22 14:36:40 +00:00
make deps.get Fetch locked version of non-forked deps
2021-05-22 16:01:58 +00:00
make dep.clone.local Clone a git dep and use the local version, eg: `make dep.clone.local dep=bonfire_me repo=https://github.com/bonfire-networks/bonfire_me`
2021-05-22 14:36:40 +00:00
make deps.clone.local.all Clone all bonfire deps / extensions
make dep.go.local~% Switch to using a local path, eg: make dep.go.local~pointers
make dep.go.local.path Switch to using a local path, specifying the path, eg: make dep.go.local dep=pointers path=./libs/pointers
2021-05-22 16:01:58 +00:00
make dep.go.git Switch to using a git repo, eg: make dep.go.git dep=pointers repo=https://github.com/bonfire-networks/pointers (specifying the repo is optional if previously specified)
make dep.go.hex Switch to using a library from hex.pm, eg: make dep.go.hex dep=pointers version="~> 0.2" (specifying the version is optional if previously specified)
2021-05-22 14:36:40 +00:00
make dep.hex~% add/enable/disable/delete a hex dep with messctl command, eg: `make dep.hex.enable dep=pointers version="~> 0.2"
make dep.git~% add/enable/disable/delete a git dep with messctl command, eg: `make dep.hex.enable dep=pointers repo=https://github.com/bonfire-networks/pointers#main
make dep.local~% add/enable/disable/delete a local dep with messctl command, eg: `make dep.hex.enable dep=pointers path=./libs/pointers
make messctl~% Utility to manage the deps in deps.hex, deps.git, and deps.path (eg. `make messctl~help`)
2021-07-03 08:04:49 +00:00
2021-05-22 14:36:40 +00:00
make contrib.forks Push all changes to the app and extensions in ./forks
make contrib.release Push all changes to the app and extensions in ./forks, increment the app version number, and push a new version/release
make contrib.app.up Update ./deps and push all changes to the app
make contrib.app.release Update ./deps, increment the app version number and push
make git.forks.add Run the git add command on each fork
make git.forks~% Run a git command on each fork (eg. `make git.forks~pull` pulls the latest version of all local deps from its git remote
2021-07-03 08:04:49 +00:00
2021-05-22 14:36:40 +00:00
make test Run tests. You can also run only specific tests, eg: `make test only=forks/bonfire_social/test`
make test.stale Run only stale tests
make test.remote Run tests (ignoring changes in local forks)
make test.watch Run stale tests, and wait for changes to any module's code, and re-run affected tests
make test.db.reset Create or reset the test DB
2021-07-03 08:04:49 +00:00
2021-05-22 14:36:40 +00:00
make rel.build Build the Docker image using previous cache
make rel.tag.latest Add latest tag to last build
make rel.push Add latest tag to last build and push to Docker Hub
make rel.run Run the app in Docker & starts a new `iex` console
make rel.run.bg Run the app in Docker, and keep running in the background
make rel.stop Run the app in Docker, and keep running in the background
make rel.shell Runs a simple shell inside of the container, useful to explore the image
2021-07-03 08:04:49 +00:00
2021-05-22 14:36:40 +00:00
make services Start background docker services (eg. db and search backends). This is automatically done for you if using Docker.
make build Build the docker image
make cmd~% Run a specific command in the container, eg: `make cmd-messclt` or `make cmd~time` or `make cmd~echo args=hello`
make shell Open the shell of the Docker web container, in dev mode
make mix~% Run a specific mix command, eg: `make mix~deps.get` or `make mix~deps.update args=pointers`
make mix.remote~% Run a specific mix command, while ignoring any deps cloned into ./forks, eg: `make mix~deps.get` or `make mix~deps.update args=pointers`
make deps.git.fix Run a git command on each dep, to ignore chmod changes
make git.merge~% Draft-merge another branch, eg `make git-merge-with-valueflows-api` to merge branch `with-valueflows-api` into the current one
make git.conflicts Find any git conflicts in ./forks
2021-05-22 14:41:05 +00:00
```
2021-05-22 14:36:40 +00:00
## Troubleshooting
### EACCES Permissions Error
If you get a permissions error when following any of the steps, run the following command and it should be fixed:
```shell
cd bonfire && sudo chown -R yourusername:yourusername .
```
Note that the command should be modified so your shell is pointing to wherever you have bonfire installed. If you are already in the bonfire directory then you only need to worry about running the `chown` portion of the command.
### Unable to access Postgres database
If you are getting any `:nxdomain` errors, check if you have any firewalls that may be blocking the port on your system.
For example, if you are running UFW (a lot of Linux distros do), run the following command to allow access to port 4000:
```shell
sudo ufw allow 4000
```
2021-05-22 14:36:40 +00:00
### (Mix) Package fetch failed
Example:
```
** (Mix) Package fetch failed and no cached copy available (https://repo.hex.pm/tarballs/distillery-2.0.12.tar)
```
In this case, distillery (as an example of a dependency) made a new release and retired the old release from hex. The new version (`2.0.14`) is quite close to the version we were depending on (`2.0.12`), so we chose to upgrade:
```shell
mix deps.update distillery
```
This respects the version bounds in `mix.exs` (`~> 2.0`), so increment that if required.
### `(DBConnection.ConnectionError) tcp recv: closed`
Example:
```
** (DBConnection.ConnectionError) tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
```
In this case, the seeds were unable to complete because a query took too long to execute on your machine. You can configure the timeout to be larger in the `dev` environment:
1. Open `config/dev.exs` in your editor.
2. Find the database configuration (search for `Bonfire.Repo`).
3. Add `timeout: 60_000` to the list of options:
```
config :bonfire, Bonfire.Repo,
timeout: 60_000,
[...]
```