Internals of Module Management in Ballerina

Published in

Ballerina Swan Lake Tech Blog

10 min readDec 23, 2018

When it comes to learning how to code in ballerina, it is important to know the internals of module management i.e. how to structure Ballerina programs, how to compile the Ballerina code, share it via repositories and how the system repository works, etc.

Stand-alone Source

Stand-alone source is a single Ballerina source file
Any valid Ballerina code can be placed inside the Ballerina source file
Ballerina code can be placed in any folder called the ‘source root’. This ‘source root’ must not contain a sub directory named ‘.ballerina’
Ballerina source files that are in the source root are in the ‘unnamed’ module.
Sub directories of the ‘source root’ do not have any semantic meaning i.e. they are not considered as modules nor as any sibling files

How can you run a Ballerina source file?

You can run a Ballerina source file in the unnamed module as follows. Lets assume that this source file has a ‘main()’ or one or more hosted services. The directory where the file resides becomes the source root. All imports in will be resolved relative to this source root.

Give the full path to the Ballerina source file if you are running it from another directoryballerina run usr/sources/foo.balGive the name of the Ballerina source file if you are running it from inside the directory
cd usr/sourcesballerina run foo.balAlter the location of the source root that the Ballerina source file will be inferred from.ballerina run --sourceroot /usr/sources/ foo.bal

You can also run any public function in the Ballerina source file as follows:

ballerina run <path-to-the-source-file>:<public-function-name>Invoke the public function 'sayHello' in the foo.balballerina run usr/sources/foo.bal:sayHello

How can you compile code in a Ballerina source file?

You can compile the code inside a Ballerina program as follows. The default file name of the compiled executable created will be the last part of module name or the Ballerina source file name (without the extension) with the extension ‘.balx’.

Give the full path to the Ballerina source file if you are compiling it from another directoryballerina build usr/sources/foo.bal [-o output-file.balx]Give the name of the Ballerina source file if you are compiling it from inside the directory
cd usr/sourcesballerina build foo.bal [-o output-file.balx]

How can you execute a compiled Ballerina program?

A compiled Ballerina program with the ‘.balx’ extension can be executed as follows:

ballerina run compiled-executable.balx

Ballerina Project

A Ballerina project can be created by executing the ‘init’ command. The init command will initialize a project with a module inside of it. If the folder where the init command is executed already contains Ballerina source files or sub directories, then they will be placed inside the new project. You can override the default files and modules created from the init command by running it in the interactive mode.
A directory is considered to be a Ballerina project if it has a sub directory called ‘.ballerina’. The ‘.ballerina’ folder is defined as the project repository.
A project will contain the following:

A user-managed project manifest file ‘Ballerina.toml’
A Ballerina-managed ‘.ballerina’ folder which is the project repository, a project-specific repository for storing module binaries
One or more modules which contain Ballerina source files

All modules in the same project will be built and assigned the same org-name and version (and other assigned metadata including dependencies). If a module in a project needs to have different versions or dependency definitions, it should be placed in its own project.
Each sub directory in the project’s root folder will be considered as a single module except for the ‘tests’ and ‘resources’ which are reserved folders.
Each sub directory name will be used as the module name. Additional sub directories within the module have no semantic meaning. All Ballerina source files inside any module sub directory will be compiled and will belong to the same module.
The compiled binaries of the modules are installed into the project repository i.e. ‘.ballerina’ after they are built.
The compiled executable of the modules are created in the ‘target’ folder
If a module is to be installed into the home repository or Ballerina Central, it will be installed from the project repository.
Ballerina source files specified in the project root are considered to be a part of the unnamed module. Each of them are compiled into the ‘target/<file-name>.balx’. This structure is not recommended to be used by the developers instead they are enforced to place their source files inside modules.
Following is a sample project structure:


  .gitignore
  Ballerina.toml       // Project Manifest
  .ballerina/          // Project repository. The project repository  
                          contains compiled module binaries
    foo.balo

  main.bal             // Part of the 'unnamed' module, compiled  
                          into main.balx in the target
                      
  foo/                 // Source in this module will be named as
                           '<org-name>/foo'
    Module.md          // Optional, contains metadata for display at 
                          Ballerina Central
    *.bal              // Ballerina source files in this directory 
                          and recursively in sub directories except 
                          tests/ and resources/
    [tests/]           // Unit and integration tests of the module 
    [resources/]       // Resources of the module

 
  target/              // Compiled executable and other artifacts
      main.balx
      Ballerina.lock   // Generated during build and is used to 
                          rebuild with the identical dependencies

The project manifest ‘Ballerina.toml’ can have the following attributes:

[project]
org-name = "natasha"
version = "1.2.0"
license = "Apache-2.0"
authors = ["Natasha"]
repository = "https://github.com/natasha/foo"
keywords = ["Natasha", "Foo", "endpoint"]# Version dependencies of the imported modules
[dependencies."natasha/firstMod"] 
version = "1.0.0"[dependencies."natasha/secMod"] 
version = "1.2.0"

If the project manifest is present, build command will inherit the execution properties from the manifest:

To determine the complete list of dependencies, all modules and source files in the project will be scanned. Then a dependency graph will be built.
All module dependencies will be checked against the versions defined in the project manifest and within the project repository.
Dependencies that are not discovered will be searched using dependency resolution in the project repository, home repository, system repository and then in Ballerina Central.
If the module dependencies are not found then the build is interrupted and error message are show to the user with a list of all missing dependencies.
Dependency modules pulled from Ballerina Central are installed into the caches repository (‘.ballerina/caches’)
Modules built will be installed into the project repository. They will not be placed in the target folder.

Organization

An organization will group modules together under a common namespace within a repository.
All modules installed into a repository must have an organization name.
Organization names can only contain lowercase alphanumeric and underscores. The maximum length is 256 characters. None of these characters have any semantic meaning.
A fully qualified module is identified within a repository by the following:

<org-name>/<module-name>[:<version>]

‘ballerina’ and ‘ballerinax’ organization names are reserved for private use. Modules in the ‘ballerina’ and ‘ballerinax’ organization will be sourced from the system repository.

Repositories

A repository is a collection of compiled Ballerina modules which can be reused and shared among other modules. There are four kinds of repositories:

Home Repository

Resides in a user’s local machine at the location of ‘BALLERINA_HOME_DIR’ or ‘~\.ballerina’ if not specified.

Project Repository

Resides within a project’s ‘.ballerina’ folder. This repository contains compiled binaries which are installed versions of modules from the project.

System Repository

Reserved repository that is placed within the Ballerina distribution. This repository contains core modules of Ballerina which have ‘ballerina’ or ‘ballerinax’ as the organization name.

Ballerina Central

Remote repository that is located at http://central.ballerina.io. This is a centrally managed repository which is used by the community to share, download, and publish Ballerina modules which can be reused.

Ballerina Modules

A Ballerina modules is a directory that contains Ballerina source files.
Modules should be a part of a Ballerina project for them to be compiled, built or executed.

How to specify a valid name for a module?

Module names can only contain alphanumeric, underscores and periods. The maximum length is 256 characters. Periods in module names have no semantic meaning other than the last segment being used as the default alias.

How to specify a version for a module?

Modules may or may not have versions.
A module cannot be pushed into a repository or a remote registry (Ballerina Central) without a version assigned. A version for the module can be assigned by having the module in a project with a manifest file ‘Ballerina.toml’.

The version should follow semantic versioning (SemVer)version = “1.0.0”

All modules in the same project will inherit the version specified in the project manifest file ‘Ballerina.toml’ i.e. all modules of the same project will have the same version.

How to import modules?

A module can be imported from a Ballerina source file which is a part of a project or just a program directory as follows:

import [<org-name>/]<pkg-name> [as identifier]];import natasha/foo; // Import a moduleimport foo;       // Import a module from the same projectimport natasha/foo as bar; // Import a module with an alias

If an<org-name> is not specified in the import, then the module must be in the same project. If the module is cannot be found in the same project, then an error will be thrown since modules will not be searched in other repositories if an org-name is not specified.

How to manage version dependencies of imported modules?

Versions for the imported modules cannot be specified in the source code. Instead the version dependencies of the imported modules within a project can be specified in the project manifest file ‘Ballerina.toml’.

[dependencies."natasha/foo"] 
version = "1.2.0"

If there is no project, then the latest module version from a repository will be used by the compiler.
If there is a project and if a version is not specified in the manifest for the imported modules, then the ‘latest’ module version in the home repository will be used by the compiler.

What is a compiled module?

A compiled module is the compiled representation of a single module of Ballerina code, without including transitive dependencies into the compiled unit.

How can you compile and build modules of a project?

All modules in the same project can be built as follows:

ballerina build

A single module in a project can be built as follows:

ballerina build <module-name>

Default name of the output will be the last part of the module name (without the extension) with the extension ‘.balo’.
When built the modules are automatically installed into the project repository, a local repository within the project directory (‘.ballerina’).

How can you install modules to your home repository?

All modules in a project can be installed to your home repository as follows:

ballerina installORballerina push --repository home

A single module in a project can be installed to your home repository as follow:

ballerina install <module-name>ballerina push <module-name> --repository home

How can you remove/uninstall modules installed to your home repository?

A module that was installed to your home repository can be removed as follows:

ballerina uninstall <org-name>/<module-name>:<version>

How to run program defined in a module?

A module in a project can be executed as follows. The run command should be executed from the project directory. Lets assume that module ‘foo’ has a ‘main()’ or one or more hosted services.

ballerina run <module-name>ballerina run foo

You can also run any public function in a module as follows:

ballerina run <module-name>:<public-function-name>Invoke the public function 'sayHello' in module 'foo'ballerina run foo:sayHello

How to install modules to a remote registry i.e. Ballerina Central?

Pushing a module uploads and installs the module into a remote registry.
The default remote registry is Ballerina Central.
You have to have a valid account token to push modules into the remote repository. If you do not have an account in Ballerina Central when you are pushing a module for the first time, the push command will redirect you to create a new account in Ballerina Central. After which the account token will be used to authenticate any module you push.
This account token should be placed in ‘Settings.toml’ which resides in your home repository at the location of ‘BALLERINA_HOME_DIR’ or ‘~\.ballerina’ if not specified. The account token is specified as follows:

[central] 
accesstoken="xxx-xxx-xxx"

If your internet is connected via a HTTP proxy, the following section has to be added to ‘Settings.toml’ to enable it.

[proxy]
host = "localhost"
port = "9494"
username = "admin"
password = "123"

All modules in a project can be pushed as follows. The project repository will be looked up to find the artifacts to be pushed.

ballerina push

A modules in a project can be pushed as follows.

ballerina push <module-name>

The organization name found in the project manifest ‘Ballerina.toml’ MUST match the account name that a user has at Ballerina Central, otherwise the command will fail. This matching criteria is an important part of making the sharing nature of modules simpler.
A module must have a version to be pushed into a remote registry, otherwise they will fail.
You cannot push a module with the same version more than once to Ballerina Central. Module versions must always be incrementally versioned when pushing to Ballerina Central. The push command will throw an error if the version hasn’t been incremented.
When pushing a module to Ballerina Central the meta data of the module which is needed for display should be specified in ‘Module.md’ which resides within the module. The first line specified in ‘Module.md’ should be less 50 characters. A sample ‘Module.md’ is as follows:

Connects to Foo from Ballerina. 

# Module Overview

The Foo connector allows you to search for public posts/statuses using the Foo REST API.You can also retrieve a status and top trends.

**Search Operations**

The `natasha/foo` module contains operations that search for public posts and statuses. **Trends Operations**

The `natasha/foo` module contains operations that retrieve top trends by location.## Compatibility
|                    |    Version     |  
|:------------------:|:--------------:|
| Ballerina Language |   0.990.0      |
| Foo API            |   1.1          |


## Sample

First, import the `natasha/foo` module into the Ballerina project.

```ballerina
import natasha/foo;
```
    
The Foo connector can be instantiated using the access token and access token secret in the Foo client config.

How to install modules into the home repository from a remote registry?

Pulling a module discovers and downloads the module source and binaries from the remote registry and installs it into the local home repository.
The default remote registry is Ballerina Central.
The pull will not repeat downloading a module if the home repository already contains it .
A module can be downloaded from Ballerina Central as follows. The downloaded module will be placed in the global cache that is within the home repository.

ballerina pull <org-name>/<module-name>[:<version>]

If a version is not specified in the pull command, it will download the latest version of the specified module from Ballerina Central to the global cache.

ballerina pull natasha/foo // Pull the latest version of the foo   
                              module availableballerina pull natasha/foo:1.0.0 // Pull the foo module of version 
                                    1.0.0

How to search and discover modules located in a remote repository?

Searching for a module searches for information about modules located in remote repositories i.e. Ballerina Central.
Information about a module which resides in a remote repository can be searched as follow:

ballerina search <text>

More information about the commands related to module management can be found here.

So this was just a brief overview about the internals of module management in Ballerina. You can try these out by downloading ballerina.

Now as for how to end with a bang? Well, that’s another blog post for another time :) :) Till we meet again :)