Skip to content
Snippets Groups Projects
Commit 9bd5d4b6 authored by Matthias Bernt's avatar Matthias Bernt
Browse files

Merge branch 'spellcheck' into 'main'

Add spellchecking

See merge request !5
parents fb31ccae bf22a328
No related branches found
No related tags found
1 merge request!5Add spellcking
Pipeline #473051 passed
stages:
- spellcheck
aspell:
stage: spellcheck
image: alpine:latest
before_script:
# Install aspell and required language dictionary
- apk add --no-cache aspell aspell-en
script:
# Find and spellcheck all markdown files
- |
#!/bin/bash
ERRORS=0
find . -name "*.md" -print0 | while IFS= read -r -d '' file; do
SPELL_OUTPUT=$(aspell --lang=en --mode=markdown --add-wordlists=custom.dict list < "$file" | tee -a spelling | sort -u)
if [ -n "$SPELL_OUTPUT" ]; then
echo "Spelling errors detected in $file"
echo "$SPELL_OUTPUT"
else
echo "No spelling errors in $file"
fi
done
if [ -n "$(cat spelling)" ]; then
echo "Spelling errors were detected. Failing the job."
exit 1
else
echo "All files passed the spellcheck!"
fi
rules:
- if: '$CI_PIPELINE_SOURCE == "push" || $CI_PIPELINE_SOURCE == "merge_request_event"'
allow_failure: false
......@@ -2,7 +2,7 @@
## Problem
Anaconda, Inc. has (rather silently) changed the terms of service ([TOS](https://www.anaconda.com/pricing/terms-of-service-faqs)) for the `defaults` channel in 2020. This means that institutions with more than 200 employes (including academic) have to pay license fees if they make use of the `defaults` channel (which might be in the worst case 50$ per month $\times$ employees since the change of TOS). Anaconda, Inc. also started to actively enforce the new policy, e.g. [against Intel](https://www.courtlistener.com/docket/69029637/anaconda-inc-v-intel-corporation/).
Anaconda, Inc. has (rather silently) changed the terms of service ([TOS](https://www.anaconda.com/pricing/terms-of-service-faqs)) for the `defaults` channel in 2020. This means that institutions with more than 200 employees (including academic) have to pay license fees if they make use of the `defaults` channel (which might be in the worst case 50$ per month $\times$ employees since the change of TOS). Anaconda, Inc. also started to actively enforce the new policy, e.g. [against Intel](https://www.courtlistener.com/docket/69029637/anaconda-inc-v-intel-corporation/).
This raised some uncertainty regarding the use of conda.
......@@ -16,7 +16,7 @@ Bottom line is: Make sure that the `defaults` channel is not used and use other
Likely the simplest advice is to double check which packages are installed. This will protect you even from misconfigurations.
- Before even trying to install the software `conda`/`mamba` update the channel information (`conda` even explicitely shows the used channels as first output). Check the the `defaults` channel is not among the channels.
- Before even trying to install the software `conda`/`mamba` update the channel information (`conda` even explicitly shows the used channels as first output). Check the the `defaults` channel is not among the channels.
- During installation the conda packages that will be installed will be listed. In the second column of this listing you find detailed information in the form `CHANNEL/PLATFORM::PACKAGE`. It seems to be good practice to double check this listing. Alternatively one could use the `--dry-run` argument.
- `conda` environments can be created from `environment.yaml` files. These files include a `channels` list. Even if a properly configured conda installation (or arguments) should ensure that the `defaults` channel is not used it seems advisable to remove the `defaults` channel from this list if it is included.
......@@ -34,7 +34,7 @@ In case installation still uses the `defaults` channel double check your configu
In order to remove an old conda distribution (like miniconda/anaconda) you need to:
- Remove the installation directory (e.g. `~/miniconda3/`). Note that this typically includes your environments which you might want to keep (see below).
- Make sure that your shell's configuration file does not contain any traces of the old conda distribution, i.e. a block starting with `# >>> conda initialize >>>` and ending with `# <<< conda initialize <<<` or any `export` statements or modifications of the `PATH` variable (only those that add conda paths). Potentially this can be done auromatically with `conda init --reverse`.
- Make sure that your shell's configuration file does not contain any traces of the old conda distribution, i.e. a block starting with `# >>> conda initialize >>>` and ending with `# <<< conda initialize <<<` or any `export` statements or modifications of the `PATH` variable (only those that add conda paths). Potentially this can be done automatically with `conda init --reverse`.
- The official documentation [also contains a guide to transitioning from `defaults` to `conda-forge`](https://conda-forge.org/docs/user/transitioning_from_defaults/)
### Check existing `conda` environments
......@@ -49,7 +49,7 @@ In order to remove an old conda distribution (like miniconda/anaconda) you need
- `mamba` does not default to the `defaults` channel.
- `conda` does not default to the `defaults` channel when installed via [miniforge](https://github.com/conda-forge/miniforge/blob/e733f7bbc41f42551e9f02766d8a0301b72fde26/README.md?plain=1#L7-L8)
- Independent of the choice you need to double check the configurations if you have previously used `conda` / `mamba` or you install conda environments using exported yaml files.
- Nowadays most features of `conda` (in particular the solver) have been intregrated in `mamba`.
- Nowadays most features of `conda` (in particular the solver) have been integrated in `mamba`.
- Nowadays `conda` and `mamba` are nearly equivalent, i.e. `mamba` covers most of `conda`'s functionality and `conda` uses `mamba`'s solver by default. Some aspects of `mamba` might be slightly faster. You can use aliases to retain script compatibility.
### Check your configuration
......@@ -59,14 +59,14 @@ In order to remove an old conda distribution (like miniconda/anaconda) you need
- Make sure that `override_channels_enabled` is enabled: `conda config --show override_channels_enabled` should show `True`.
If needed set it with `conda config --set override_channels_enabled True`.
- It might be also a good idea to add [`nodefaults`](https://docs.conda.io/projects/conda/en/4.6.1/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually) to the channel list, even if it's documented as equivalent to `override_channels_enabled`
- If prefered one can specify `--override-channels --channel conda-forge` for `conda install` / `conda create` to achieve the same effect. Additional channels by be added by listing more `--channel ZYZ` arguments.
- If preferred one can specify `--override-channels --channel conda-forge` for `conda install` / `conda create` to achieve the same effect. Additional channels by be added by listing more `--channel ZYZ` arguments.
## Details and background
Software (and in particular scientific software) is often difficult to install. Additionally, reproducible science requires independent installations of multiple versions of a software. `anaconda` aims to provide easy installation of important scientific software with proper version management. To this end the `conda` package manager has been created. It provides the possibility to install software independent of the programming language (it covers much more than only python and R) and operating system (it covers Linux, Mac and Windows).
At the source of the conda ecosystem there are conda *recipes* which contain information on metadata, requirements and installation instructions that are needed to build a package. These `packages` which are a pre-built archives containing software, metadata, and information on dependencies, ready to be installed into a Conda environment. These packages are stored in
`channels`, i.e. repositories (online or local) where conda packages are stored and made available for installation. Many channels are free and contain only free and open source software - most importnatly `conda-forge` and `bioconda`.
`channels`, i.e. repositories (online or local) where conda packages are stored and made available for installation. Many channels are free and contain only free and open source software - most importantly `conda-forge` and `bioconda`.
The `defaults` channel consists of conda packages that are maintained by Anaconda, Inc. It consists of several [subchannels](https://docs.anaconda.com/working-with-conda/reference/default-repositories/): `pkgs/main`, `pkgs/r` `pkgs/pro`, `pkgs/msys2`, `pkgs/free`, `pkgs/archive`.
Most conda channels are currently hosted by Anaconda Inc (if one is looking for risks in using `conda` then this might be one since its a single point of failure).
Locally conda `packages` can be installed in environments which will install the package and its requirements.
......@@ -84,10 +84,10 @@ It seems to be a good idea to rely on community maintained channels that include
### The larger picture
Beyond the problem with Anaconda's `defaults` channel, it is worth to mention that currently reproducible science relies to a significant extend on the abily to freely use services offered by companies, see discussion [here](https://galaxyproject.org/news/2024-08-20-opinion-conda/).
Beyond the problem with Anaconda's `defaults` channel, it is worth to mention that currently reproducible science relies to a significant extend on the availability to freely use services offered by companies, see discussion [here](https://galaxyproject.org/news/2024-08-20-opinion-conda/).
It's easy to blame Anaconda Inc for the TOS change, but maybe a bit of appreciation might be appropriate, but communication certainly [might have been better](https://www.theregister.com/2024/08/08/anaconda_puts_the_squeeze_on/).
Also note that Anaconda Inc. continues to [provide substantial ressources](https://galaxyproject.org/news/2024-08-20-opinion-conda/) for free to everyone (and in particular the scientific community).
Also note that Anaconda Inc. continues to [provide substantial resources](https://galaxyproject.org/news/2024-08-20-opinion-conda/) for free to everyone (and in particular the scientific community).
- The software `conda` (which is also maintained by Anaconda, Inc) is still ([and will always be](https://www.anaconda.com/pricing/terms-of-service-faqs)) free to use and [open source](https://github.com/conda/conda/blob/main/LICENSE).
- Hosting of the packages of all `conda` channels. The open source community currently has no possibilities to host these packages elsewhere ([see](https://conda-forge.org/blog/2020/11/20/anaconda-tos/)).
......@@ -98,13 +98,13 @@ Note that Helmholtz (HiFiS) made some important steps toward scientific infrastr
### Discussion of other documents
Recently [UFZ](https://www.intranet.ufz.de/index.php?de=31339&nb_item=2803) and [HiFiS](https://www.hifis.net/news/2024/10/15/anaconda-licensing.html) pubished recommendations.
Recently [UFZ](https://www.intranet.ufz.de/index.php?de=31339&nb_item=2803) and [HiFiS](https://www.hifis.net/news/2024/10/15/anaconda-licensing.html) published recommendations.
While these documents are certainly important to raise awareness, they contain a few points that are worth discussing
> The company Anaconda provides very convenient software development environments and libraries for Python and R, especially for beginners. Due to their many years of free availability for research and educational institutions, they are also very popular in training and in the discussion on StackOverflow.
Partially correct, since the software is not provided by Anaconda, but only distributed. All (or allmost all) software that is distributed in the `defaults` channel is FOSS software and distributed in many other (usually free) ways, e.g. `conda-forge`.
Partially correct, since the software is not provided by Anaconda, but only distributed. All (or almost all) software that is distributed in the `defaults` channel is FOSS software and distributed in many other (usually free) ways, e.g. `conda-forge`.
Furthermore, it's important to highlight that these statements only apply to the software in the `defaults` channel. The possibilities offered by `conda` and channels like `conda-forge` go far beyond this.
- Proper versioned software environments for all sorts of interpreted and compiled software.
......@@ -122,7 +122,7 @@ As discussed above, with minimal effort, both can be used without using the prob
This mixes two different concepts. `miniforge` is a conda distribution and `conda-forge` a conda channel. Both are part of the solution.
All other recommentations only apply to python (and are therefore no full alternative) or are unrelated to the actual question.
All other recommendations only apply to python (and are therefore no full alternative) or are unrelated to the actual question.
Also the Helmholtz Open Science newsletter recently [talked about this topic](https://os.helmholtz.de/en/newsroom/newsletter/106th-newsletter/#c124408)
......@@ -131,4 +131,4 @@ Also the Helmholtz Open Science newsletter recently [talked about this topic](ht
In fact all software that is distributed in Anaconda's `defaults` channel is FOSS software. The channel is only a non-free means of distributing the software in a way that is much more convenient than anything before `conda` was available.
Nowadays, with `conda-forge` equally convenient and free ways are available.
The Jülich Supercomputing Centre's RSE Team has also [uploaded a recommendation](https://www.fz-juelich.de/en/rse/the_latest/the-anaconda-is-squeezing-us). Notably, the JSC typically encourages the use of virtual environments that are implemented in Python rather than a package manager such as `conda`, as it is better compatible with EasyBuild (including user modules).
\ No newline at end of file
The Jülich Supercomputing Centre's RSE Team has also [uploaded a recommendation](https://www.fz-juelich.de/en/rse/the_latest/the-anaconda-is-squeezing-us). Notably, the JSC typically encourages the use of virtual environments that are implemented in Python rather than a package manager such as `conda`, as it is better compatible with EasyBuild (including user modules).
conda
Conda's
EasyBuild
FOSS
HiFiS
installable
JSC
Jülich
miniconda
miniforge
Miniforge
misconfigurations
pre
RSE
StackOverflow
subchannels
Supercomputing
TOS
UFZ
yaml
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment