Update README.org
This commit is contained in:
EatThePooh
GitHub
parent
bbee313d53
commit
6ce4d88ac3
1 changed files with 186 additions and 108 deletions
294
README.org
294
README.org
|
|
@ -1,139 +1,177 @@
|
|||
:PROPERTIES:
|
||||
:ID: 55a85682-80d7-41fe-ab55-d3afe9abbc38
|
||||
:END:
|
||||
#+title: Grimu-R
|
||||
Grimu-R is an open ecosystem and a framework for mixed visual-textual dataflow scripting.
|
||||
|
||||
** Overview :noexport:
|
||||
Grimu-R is an open ecosystem and a framework for mixed visual-textual dataflow scripting. It consists primarily of the following components:
|
||||
It consists primarily of the following components:
|
||||
- Execution model allowing computations to be expressed as reactive graphs
|
||||
- Polyglot scripting runtime environment
|
||||
- Live web-based visual-textual editor
|
||||
** Table of Contents :TOC_3_gh:
|
||||
- [[#goals][Goals]]
|
||||
- [[#immediate-feedback-and-composability][Immediate feedback and composability]]
|
||||
- [[#executable-build-targets][Executable build targets]]
|
||||
- [[#hassle-free-reproducible-builds][Hassle-free reproducible builds]]
|
||||
- [[#reactivity-taken-seriously][Reactivity taken seriously]]
|
||||
- [[#malleability][Malleability]]
|
||||
- [[#unlimited-extensibility][Unlimited extensibility]]
|
||||
- [[#powerful-out-of-the-box-toolkit][Powerful out-of-the-box toolkit]]
|
||||
- [[#non-goals][Non-goals]]
|
||||
- [[#use-cases][Use cases]]
|
||||
- [[#positioning][Positioning]]
|
||||
- [[#nomenclature][Nomenclature]]
|
||||
- [[#weave][Weave]]
|
||||
- [[#ornament][Ornament]]
|
||||
- [[#spell][Spell]]
|
||||
- [[#scroll][Scroll]]
|
||||
- [[#inscription][Inscription]]
|
||||
- [[#how-it-works][How it works]]
|
||||
- [[#development-environment][Development environment]]
|
||||
- [[#workflow][Workflow]]
|
||||
- [[#cross-language-connections][Cross-language connections]]
|
||||
- [[#execution-environment][Execution environment]]
|
||||
- [[#example-project-structure][Example project structure]]
|
||||
- [[#faq][FAQ]]
|
||||
- [[#is-this-a-yet-another-web-framework][Is this a yet another web framework?]]
|
||||
- [[#a-user-can-execute-arbitrary-code-on-my-backend-really][A user can execute arbitrary code on my backend? Really?]]
|
||||
- [[#can-i-opt-out-of-nix][Can I opt out of Nix?]]
|
||||
- [[#how-do-i-test-my-weave][How do I test my weave?]]
|
||||
- [[#features][Features]]
|
||||
- [[#inspiration][Inspiration]]
|
||||
* Table of Contents :TOC_2_gh:
|
||||
- [[#inspiration][Inspiration]]
|
||||
- [[#goals][Goals]]
|
||||
- [[#immediate-feedback-and-composability][Immediate feedback and composability]]
|
||||
- [[#executable-build-targets][Executable build targets]]
|
||||
- [[#hassle-free-reproducible-builds][Hassle-free reproducible builds]]
|
||||
- [[#reactivity-taken-seriously][Reactivity taken seriously]]
|
||||
- [[#malleability][Malleability]]
|
||||
- [[#unlimited-extensibility][Unlimited extensibility]]
|
||||
- [[#powerful-out-of-the-box-toolkit][Powerful out-of-the-box toolkit]]
|
||||
- [[#non-goals][Non-goals]]
|
||||
- [[#use-cases][Use cases]]
|
||||
- [[#positioning][Positioning]]
|
||||
- [[#nomenclature][Nomenclature]]
|
||||
- [[#weave][Weave]]
|
||||
- [[#ornament][Ornament]]
|
||||
- [[#spell][Spell]]
|
||||
- [[#scroll][Scroll]]
|
||||
- [[#inscription][Inscription]]
|
||||
- [[#how-it-works][How it works]]
|
||||
- [[#development-environment][Development environment]]
|
||||
- [[#workflow][Workflow]]
|
||||
- [[#cross-language-connections][Cross-language connections]]
|
||||
- [[#execution-environment][Execution environment]]
|
||||
- [[#features][Features]]
|
||||
- [[#faq][FAQ]]
|
||||
- [[#is-this-a-yet-another-web-framework][Is this a yet another web framework?]]
|
||||
- [[#a-user-can-execute-arbitrary-code-on-my-backend-really][A user can execute arbitrary code on my backend? Really?]]
|
||||
- [[#can-i-opt-out-of-nix][Can I opt out of Nix?]]
|
||||
- [[#how-do-i-test-my-weave][How do I test my weave?]]
|
||||
- [[#can-i-intergate-ai-models][Can I intergate AI models?]]
|
||||
|
||||
* Inspiration
|
||||
- [[https://github.com/enso-org/enso][Enso]]
|
||||
- [[https://unit.software][Unit]]
|
||||
- [[https://jupyter.org/][Jupyter Notebooks]]
|
||||
- [[https://observablehq.com/][Observable]]
|
||||
* Goals
|
||||
** Immediate feedback and composability
|
||||
|
||||
Grimu-R makes it easy to inject new processing steps, visualize intermediate results,
|
||||
experiment and reuse previous work.
|
||||
|
||||
** Executable build targets
|
||||
|
||||
** Goals
|
||||
*** Immediate feedback and composability
|
||||
Grimu-R makes it easy to inject new processing steps, visualize intermediate results, experiment and reuse previous work.
|
||||
*** Executable build targets
|
||||
A live visual editor allows one to prototype, debug and execute one-off jobs comfortably.
|
||||
|
||||
But the work can always be distilled into an executable artifact, such as: a CLI tool, an RPC server, or a web application, to be distributed as such or integrated into larger workflows.
|
||||
*** Hassle-free reproducible builds
|
||||
But the work can always be distilled into an executable artifact, such as:
|
||||
a CLI tool, an RPC server, or a web application, to be distributed as such or integrated into larger workflows.
|
||||
|
||||
** Hassle-free reproducible builds
|
||||
|
||||
Powered by Nix, Grimu-R projects require minimal effort to build, run, or depend on.
|
||||
*** Reactivity taken seriously
|
||||
Push or pull? Throttle, debounce, batch? Granular configuration lets you decide precisely how reactivity works in your dataflows, giving you a way to manage backpressure, retries and failures.
|
||||
*** Malleability
|
||||
End users of Grimu-R applications enjoy the same superpowers as their developers, being able to inspect or modify the dataflow graph and its constituents using the same editor (with caveats).
|
||||
*** Unlimited extensibility
|
||||
|
||||
** Reactivity taken seriously
|
||||
|
||||
Push or pull? Throttle, debounce, batch?
|
||||
|
||||
Granular configuration lets you decide precisely how reactivity works in your dataflows,
|
||||
giving you a way to manage backpressure, retries and failures.
|
||||
|
||||
** Malleability
|
||||
|
||||
End users of Grimu-R applications enjoy the same superpowers as their developers,
|
||||
being able to inspect or modify the dataflow graph and its constituents using the same editor (with caveats).
|
||||
|
||||
** Unlimited extensibility
|
||||
|
||||
Unlike some similar products, Grimu-R doesn't stop at ingesting CSVs and calling HTTP endpoints.
|
||||
|
||||
Real code blocks in familiar languages are prioritized from day one, leveraging their own ecosystems.
|
||||
*** Powerful out-of-the-box toolkit
|
||||
|
||||
** Powerful out-of-the-box toolkit
|
||||
|
||||
To get you started without implementing your own components.
|
||||
** Non-goals
|
||||
|
||||
* Non-goals
|
||||
Some things Grimu-R itself will not take care of, delegating them to the surrounding environment:
|
||||
|
||||
- Resumable/durable execution
|
||||
- Complex state management beyond reactive dataflows
|
||||
- Distributed computing
|
||||
- Distributed computing (outside of client-server)
|
||||
- Collaborative editing
|
||||
- Untrusted code sandboxing
|
||||
- Authentication and access control
|
||||
- Telemetry
|
||||
** Use cases
|
||||
* Use cases
|
||||
- Rapid prototyping of data processing pipelines
|
||||
- Self-hosted workflow automation
|
||||
- Dashboards with custom logic
|
||||
- Internal tooling and automation scripts
|
||||
- Hacker-friendly, highly customizable applications
|
||||
** Positioning
|
||||
* Positioning
|
||||
Grimu-R aims to take the best without the worst from:
|
||||
|
||||
- Rigid, limited or proprietary visual programming environments
|
||||
- Messy, brittle script bundles held together by spit and duct tape
|
||||
- Opaque, obscure, perishable code notebooks
|
||||
** Nomenclature
|
||||
Grimu-R introduces some fanciful jargon to avoid the wrong technical connotations. Below is a brief explanation for the key terms:
|
||||
*** Weave
|
||||
A Grimu-R project's weave is what changes when someone is working on it ("weaving"). It's a persistent representation of the dataflow graph and its graphic layout.
|
||||
*** Ornament
|
||||
Weaves are composed of ornaments, which serve the dual purpose of reusable modules and entry points. An ornament would correspond to an RPC method or a CLI subcommand. Ornaments can be nested within other ornaments, appearing as single nodes that encapsulate their internal graphs.
|
||||
*** Spell
|
||||
Spells are the basic building blocks for ornaments. A spell is just a normal computer program that acts as a data source (e. g. a static CSV sheet), data sink (e. g. a POST endpoint) or a data transformer (e. g. grep). Each spell is based on a scroll and can contain a dynamic script ("inscription") stored in the weave.
|
||||
*** Scroll
|
||||
Scrolls define a default configuration for spells, as well as the runtime environment to run a spell in. They are developed outside of Grimu-R using standard development tooling.
|
||||
*** Inscription
|
||||
Spells based on interpreter scrolls (like Bash, Node.js) can have their inscriptions edited directly in the visual editor, enabling live code changes without rebuilding the development environment.
|
||||
** How it works
|
||||
*** Development environment
|
||||
Grimu-R projects being Nix flakes, the development tooling ("loom") is provisioned as a Nix shell. It gives you the visual editor, the companion CLI tool and the scroll-defined execution context for spells.
|
||||
*** Workflow
|
||||
The visual editor is served by a development or application server. Changes and execution requests for server-side spells are sent over a bidirectional WebSocket connection. User modifications are stored locally in the browser's storage, exportable as a file. Weaves are tracked by usual VSC tools.
|
||||
*** Cross-language connections
|
||||
* Nomenclature
|
||||
Grimu-R introduces some fanciful jargon to avoid the wrong technical connotations.
|
||||
|
||||
Below is a brief explanation for the key terms:
|
||||
|
||||
** Weave
|
||||
|
||||
A Grimu-R project's weave is what changes when someone is working on it ("weaving").
|
||||
|
||||
It's a persistent representation of the dataflow graph and its graphic layout.
|
||||
|
||||
** Ornament
|
||||
|
||||
Weaves are composed of ornaments, which serve the dual purpose of reusable modules and entry points.
|
||||
|
||||
An ornament would correspond to an RPC method or a CLI subcommand.
|
||||
|
||||
Ornaments can be nested within other ornaments, appearing as single nodes that encapsulate their internal graphs.
|
||||
|
||||
** Spell
|
||||
|
||||
Spells are the basic building blocks for ornaments.
|
||||
|
||||
A spell is just a normal computer program that acts as:
|
||||
- data source (e. g. a static CSV sheet)
|
||||
- data sink (e. g. a POST endpoint)
|
||||
- data transformer (e. g. grep)
|
||||
|
||||
Each spell is based on a scroll and can contain a dynamic script ("inscription") stored in the weave.
|
||||
|
||||
** Scroll
|
||||
|
||||
Scrolls define a default configuration for spells, as well as the runtime environment to run a spell in.
|
||||
|
||||
They are developed outside of Grimu-R using standard development tooling.
|
||||
|
||||
** Inscription
|
||||
|
||||
Spells based on interpreter scrolls (like Bash, Node.js) can have their inscriptions edited
|
||||
directly in the visual editor, enabling live code changes without rebuilding the development environment.
|
||||
|
||||
* How it works
|
||||
** Development environment
|
||||
|
||||
Grimu-R projects being Nix flakes, the development tooling ("loom") is provisioned as a Nix shell.
|
||||
|
||||
It gives you the visual editor, the companion CLI tool and the scroll-defined execution context for spells.
|
||||
|
||||
** Workflow
|
||||
|
||||
- The visual editor is served by a development or application server.
|
||||
- Changes and execution requests for server-side spells are sent over a bidirectional WebSocket connection.
|
||||
- User modifications are stored locally in the browser's storage, exportable as a file.
|
||||
- Weaves are tracked by usual VSC tools.
|
||||
|
||||
** Cross-language connections
|
||||
|
||||
Think Unix pipes but with structured data formats like JSON lines instead of plain text.
|
||||
*** Execution environment
|
||||
Scrolls used in a weave define their requirements in Nix terms. For example, if several scrolls need Node.js, they can all refer to the same nodejs Nix package defined in the flake. All of that, and optionally the visual editor, is bundled into a target build ("edition").
|
||||
*** Example project structure
|
||||
- flake.nix
|
||||
- flake.lock
|
||||
- weave
|
||||
+ weave.hjson
|
||||
+ main-ornament.wiv
|
||||
+ ornament1.wiv
|
||||
+ ornament2.wiv
|
||||
- scriptorium
|
||||
+ scriptorium.nix
|
||||
+ scroll1
|
||||
* scroll.hjson
|
||||
* index.ts
|
||||
+ scroll2
|
||||
* dump.csv
|
||||
** FAQ
|
||||
*** Is this a yet another web framework?
|
||||
No, but it could be.
|
||||
|
||||
One framework called [[https://cycle.js.org/][Cycle.js]] defines a web application as a pure dataflow mapping that has sinks wired back to sources, proxied by drivers that handle side effects.
|
||||
** Execution environment
|
||||
|
||||
Grimu-R can do a similar thing if you want it to, although this usage is not supported out of the box.
|
||||
Scrolls used in a weave define their requirements in Nix terms.
|
||||
|
||||
Or you could use any framework you like with the dataflow stuff being incidental.
|
||||
*** A user can execute arbitrary code on my backend? Really?
|
||||
By default, any client-server Grimu-R build ("edition") would allow users to see and edit client-side parts of the weave only.
|
||||
For example, if several scrolls need Node.js, they can all refer to the same nodejs Nix package defined in the flake.
|
||||
All of that, and optionally the visual editor, is bundled into a target build ("edition").
|
||||
|
||||
This is but superficially different from users messing around in their browser's console or employing a browser extension.
|
||||
|
||||
On the other hand, if the server-side parts run in a trusted environment with authorized access, you could give users a more generous degree of control.
|
||||
*** Can I opt out of Nix?
|
||||
To some extent, yes. While Grimu-R is built on top of Nix and wouldn't make a lot of sense otherwise, you always have the option to access out-of-store files from your spells' inscriptions, at the cost of reproducibility.
|
||||
*** How do I test my weave?
|
||||
If you do it manually: right in the visual editor. Add spells with a constant output as test data sources and stitch them to the spells you want to test.
|
||||
|
||||
If you need automation: build an edition and call the ornaments programmatically. If you already have test data stored in the weave, you can extract it by the spell's id using loom CLI or library.
|
||||
** Features
|
||||
* Features
|
||||
| | On the roadmap | Ideas |
|
||||
|--------------------------+--------------------------------+--------------------|
|
||||
| Editions (build targets) | Self-sufficient CLI executable | Mobile application |
|
||||
|
|
@ -143,8 +181,48 @@ If you need automation: build an edition and call the ornaments programmatically
|
|||
| Scripting languages | TypeScript/JavaScript | DuckDB SQL |
|
||||
| | Bash | Python |
|
||||
| | Haskell | R |
|
||||
** Inspiration
|
||||
- [[https://github.com/enso-org/enso][Enso]]
|
||||
- [[https://unit.software][Unit]]
|
||||
- [[https://jupyter.org/][Jupyter Notebooks]]
|
||||
- [[https://observablehq.com/][Observable]]
|
||||
* FAQ
|
||||
** Is this a yet another web framework?
|
||||
|
||||
No, but it could be.
|
||||
|
||||
One framework called [[https://cycle.js.org/][Cycle.js]] defines a web application as a pure dataflow mapping that has sinks wired back to sources, proxied by drivers that handle side effects.
|
||||
|
||||
Grimu-R can do a similar thing if you want it to, although this usage is not supported out of the box.
|
||||
|
||||
Or you could use any framework you like with the dataflow stuff being incidental.
|
||||
|
||||
** A user can execute arbitrary code on my backend? Really?
|
||||
|
||||
By default, any client-server Grimu-R build ("edition") would allow users to see
|
||||
and edit client-side parts of the weave only.
|
||||
|
||||
This is but superficially different from users messing around in their
|
||||
browser's console or employing a browser extension.
|
||||
|
||||
On the other hand, if the server-side parts run in a trusted environment with authorized access,
|
||||
you could give users a more generous degree of control.
|
||||
|
||||
** Can I opt out of Nix?
|
||||
|
||||
To some extent, yes.
|
||||
|
||||
While Grimu-R is built on top of Nix and wouldn't make a lot of sense otherwise,
|
||||
you always have the option to access out-of-store files from your spells' inscriptions,
|
||||
at the cost of reproducibility.
|
||||
|
||||
** How do I test my weave?
|
||||
|
||||
If you do it manually: right in the visual editor.
|
||||
Add spells with a constant output as test data sources and stitch them to the spells you want to test.
|
||||
|
||||
If you need automation: build an edition and call the ornaments programmatically.
|
||||
If you already have test data stored in the weave, you can extract it by the spell's id using loom CLI or library.
|
||||
|
||||
** Can I intergate AI models?
|
||||
|
||||
If you want something like [[https://computer.tldraw.com/][tldraw computer]], Grimu-R would accomodate that and beyond,
|
||||
potentially supporting fancier features like tool use and MCP servers.
|
||||
|
||||
However, until someone contributes scrolls tailored for the purpose, Grimu-R would offer worse ergonomics.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue