gr-base/README.org

:PROPERTIES:
:ID:       55a85682-80d7-41fe-ab55-d3afe9abbc38
:END:
#+OPTIONS: toc:nil
#+BEGIN_EXPORT html
<style>
body {
  font-family: 'Courier New', Courier, monospace;
  background-color: ivory;
  line-height: 1.5;
  padding-right: 1em;
}

h2 {
  margin-top: 1.8rem;
  padding-left: 0.5rem;
  border-left: 4px solid #3498db;
}


table {
  width: 100%;
}

#content {
  max-width: 48em;
}

.outline-2 div {
  padding-left: 1em;
}

a:link { 
  color: #2E5EAA;
  text-decoration: none;
}
a:visited { 
  color: #663399;
}
a:hover { 
  color: #1a4480; 
  text-decoration: underline; 
}
</style>
#+END_EXPORT
#+title: Grimu-R
Grimu-R is an *open ecosystem* and a framework for mixed visual-textual *dataflow* scripting.

It consists primarily of the following components:
- Execution model allowing computations to be expressed as *reactive graphs*
- *Polyglot scripting* runtime environment
- Interactive web-based *visual-textual* editor
#+TOC: headlines 2
* Table of Contents                                                :TOC_2_gh:noexport:
- [[#inspiration][Inspiration]]
- [[#goals][Goals]]
  - [[#immediate-feedback-and-composability][Immediate feedback and composability]]
  - [[#embeddable-build-targets][Embeddable build targets]]
  - [[#hassle-free-reproducible-builds][Hassle-free reproducible builds]]
  - [[#reactivity-taken-seriously][Reactivity taken seriously]]
  - [[#malleability][Malleability]]
  - [[#unlimited-extensibility][Unlimited extensibility]]
  - [[#powerful-out-of-the-box-toolkit][Powerful out-of-the-box toolkit]]
- [[#non-goals][Non-goals]]
- [[#use-cases][Use cases]]
- [[#positioning][Positioning]]
- [[#nomenclature][Nomenclature]]
  - [[#weave][Weave]]
  - [[#ornament][Ornament]]
  - [[#spell][Spell]]
  - [[#scroll][Scroll]]
  - [[#inscription][Inscription]]
- [[#how-it-works][How it works]]
  - [[#development-environment][Development environment]]
  - [[#workflow][Workflow]]
  - [[#cross-language-connections][Cross-language connections]]
  - [[#execution-environment][Execution environment]]
- [[#features][Features]]
- [[#faq][FAQ]]
  - [[#is-this-a-yet-another-web-framework][Is this a yet another web framework?]]
  - [[#how-does-malleability-work-in-client-server-applications][How does malleability work in client-server applications?]]
  - [[#can-i-opt-out-of-nix][Can I opt out of Nix?]]
  - [[#how-do-i-test-my-weave][How do I test my weave?]]
  - [[#can-i-integrate-ai-models][Can I integrate AI models?]]

* Inspiration
- [[https://github.com/enso-org/enso][Enso]]
- [[https://unit.software][Unit]]
- [[https://jupyter.org/][Jupyter Notebooks]]
- [[https://observablehq.com/][Observable]]
* Goals
** Immediate feedback and composability

Grimu-R makes it easy to inject new processing steps, visualize intermediate results,
experiment and reuse previous work.

** Embeddable build targets

An interactive visual editor allows one to prototype, debug and run one-off jobs comfortably.

But the work can always be distilled into an executable artifact:
a CLI tool, an RPC server, or a web application, to be distributed as such or integrated into larger workflows.

** Hassle-free reproducible builds

Powered by Nix, Grimu-R projects require minimal effort to build, run, or depend on.

For interactive use with scripts, the default mode is runtime dependencies.

For builds, it is recommended to have dependencies defined statically and provisioned by Nix.

** Reactivity taken seriously

Push or pull? Throttle, debounce, batch?

Granular configuration lets you decide precisely how reactivity works in your dataflows,
giving you a way to manage backpressure, retries and failures.

** Malleability

End users of Grimu-R applications enjoy the same superpowers as their developers,
being able to inspect or modify the dataflow graph and its constituents using the same editor (with caveats).

** Unlimited extensibility

Unlike some similar products, Grimu-R doesn't stop at ingesting CSVs and calling HTTP endpoints.

Real code blocks in familiar languages are prioritized from day one, leveraging their own ecosystems.

** Powerful out-of-the-box toolkit

Get started without implementing your own components.

No code to low code to high code.

* Non-goals
Some things Grimu-R itself will not take care of, delegating them to the surrounding environment:

- Resumable/durable execution
- Distributed deployment orchestration
- Collaborative editing
- Authentication and access control
- Telemetry
* Use cases
- Rapid prototyping of data processing pipelines
- Self-hosted workflow automation
- Dashboards with custom logic
- Internal tooling and automation scripts
- Hacker-friendly, highly customizable applications 
* Positioning
Grimu-R aims to take the best without the worst from:

- Rigid, limited or proprietary visual programming environments
- Messy, brittle script bundles held together by spit and duct tape
- Opaque, obscure, perishable code notebooks

while being developer-first.
* Nomenclature
Grimu-R introduces some fanciful jargon to avoid the wrong technical connotations.

Below is a brief explanation for the key terms:

** Weave

A Grimu-R project's weave is what changes when someone is working on it ("weaving").

It's a persistent representation of the dataflow graph and its graphic layout.

** Ornament

Weaves are composed of ornaments, which serve the dual purpose of reusable modules and entry points.

An ornament would correspond to an RPC method or a CLI subcommand.

Ornaments can be nested within other ornaments, appearing as single nodes that encapsulate their internal graphs.

** Spell

Spells are the basic building blocks for ornaments.

A spell is just a normal computer program that acts as:
- data source (e. g. a static CSV sheet)
- data sink (e. g. a POST endpoint)
- data transformer (e. g. grep)

Each spell is based on a scroll and can contain a dynamic script ("inscription") stored in the weave.

** Scroll

Scrolls define a default configuration for spells, as well as the runtime environment to run a spell in.

They are developed outside of Grimu-R using standard development tooling.

** Inscription

Spells based on interpreter scrolls (like Bash, Node.js) can have their inscriptions edited
directly in the visual editor, enabling live code changes without rebuilding the development environment.

* How it works
** Development environment

Grimu-R projects being Nix flakes, the development tooling ("loom") is provisioned in a Nix shell.

It gives you the visual editor, the companion CLI tool and the execution context for spells.

** Workflow

- The visual editor is served by a development or application server.
- Changes and execution requests for server-side spells are sent over a bidirectional WebSocket connection.
- User modifications are stored locally in the browser's storage, exportable as a file.
- Weaves are tracked by usual VCS tools.

** Cross-language connections

Think Unix pipes but with structured data formats like JSON lines instead of plain text.

** Execution environment

Purpose-specific scrolls used in a weave have their requirements defined in Nix terms.

For example, if several scrolls need Deno, they can all refer to the same deno Nix package defined in the flake and declare their Deno dependencies normally.

A single copy of Deno runtime and a shared Deno cache would be bundled in an edition.

However, the general-purpose Deno scroll executing arbitrary TypeScript inscriptions
would have Deno fetch dependencies during runtime.

* Features
|                          | On the roadmap                 | Ideas              |
|--------------------------+--------------------------------+--------------------|
| Editions (build targets) | Self-sufficient CLI executable | Mobile application |
|                          | CLI client + RPC server        | React component    |
|                          | Web application                |                    |
|                          |                                |                    |
| Scripting languages      | TypeScript                     | DuckDB SQL         |
|                          | Bash                           | Python             |
|                          | Haskell                        | R                  |
* FAQ
** Is this a yet another web framework?

No, but it could be.

One framework called [[https://cycle.js.org/][Cycle.js]] defines a web application as a pure dataflow mapping that has sinks wired back to sources, proxied by drivers that handle side effects.

Grimu-R can do a similar thing if you want it to.
Or you could use any framework you like with the dataflow stuff being incidental.

** How does malleability work in client-server applications?

Unlike standalone editions executing entirely on the end user's machine,
client-server editions split the weave between runtime components.

Only the client-side part of the weave can be inspected and edited by
the end user.

** Can I opt out of Nix?

To some extent, yes.

While Grimu-R is built on top of Nix and wouldn't make a lot of sense otherwise,
you always have the option to access out-of-store files from your spells' inscriptions,
at the cost of reproducibility.

So you could use e. g. runtime JS imports or call some executable on PATH from a shell script.

** How do I test my weave?

If you do it manually: right in the visual editor.
Add spells with a constant output as test data sources and stitch them to the spells you want to test.

If you need automation: build an edition and call the ornaments programmatically.
If you already have test data stored in the weave, you can extract it by the spell's id using loom CLI or library.

** Can I integrate AI models?

There are three integration scenarios which would make sense:

1. AI spells in the weave:

   Spells calling inference APIs, equipped with MCP spells and
   template spells for prompt composition

2. AI weaving copilot:

   Loom providing AI tools to edit the weave

3. MCP server editions:

   Ornaments exposed as tools for external AI systems

#1 is easy to do even with no out-of-the-box support.

#2 and #3 would require some work on development tooling.