---

pypette (to be read as pipette) is a module which makes building pipelines ridiculously simple, allowing users to control the flow with minimal instructions.

Features

• Ridiculously simple interface.

• Ability to view pipeline structure within the comfort of a terminal.

• Run pipeline in exception resilient way if needed.

• Create dependencies on pipelines easily.

• Generate a easy to view/understand report within the comfort of a terminal.

Setup

Using pip

pip install pypette

Directly from the repository

git clone https://github.com/csurfer/pypette.git
python pypette/setup.py install

Quick Start

from pypette import BashJob, Job, Pipe

def wish():
    print("Hello World!")

wish_job = Job(wish)

build_job = BashJob(["python", "setup.py", "build"])

test_job = BashJob(["python", "setup.py", "test"])

clean_job = BashJob(["rm", "*.pyc"])

def goodbye(msg):
    print("Goodbye {}!".format(msg))

goodbye_job = Job(goodbye, args=("World",))

# Pipeline can be created using add_jobs
p = Pipe("Pipeline")
p.add_jobs([wish_job])
p.add_jobs([build_job, test_job], run_in_parallel=True)
p.add_jobs([clean_job])
p.add_jobs([goodbye_job])

# Or using add_stage
p = Pipe("Pipeline")
p.add_stage(wish_job)
p.add_stage(build_job, test_job)
p.add_stage(clean_job)
p.add_stage(goodbye_job)

# Can check how the pipeline looks like
p.graph()

# Can run the pipeline using
p.run()

# Can check on the success states of different stages of pipeline using
p.report()

Example pipeline

An example pipeline and its code are included in examples folder.

User Guide

• Job
  • for methods without arguments
  • for methods with arguments
  • for lambda methods
• BashJob
• Pipe
  • Creation Modes
  • Adding jobs to pipeline
  • Adding dependencies to pipelines
  • Visualizing the pipeline structure
  • Running the pipeline
  • Generating report for the pipeline run
• Building complex pipelines
  • Exception scenarios
  • Combining everything

API Reference

If you are looking for information on a specific function, class or method, this part of the documentation is for you.

• Gate Object
• BashJob Object
• Job Object
• Pipe Object
• ThreadState Object
• ThreadWrapper Object

Contributing

Bug Reports and Feature Requests

Please use issue tracker for reporting bugs or feature requests.

Development

Pull requests are most welcome. Kindly follow the steps suggested below:

1. Checkout the repository.

2. Make your changes and add/update relavent tests.

3. Install `poetry` using `pip install poetry`.

4. Run `poetry install` to create project’s virtual environment.

5. Run tests using `poetry run tox` (Any python versions which you don’t have checked out will fail this). Fix failing tests and repeat.

6. Make documentation changes that are relavant.

7. Install `pre-commit` using `pip install pre-commit` and run `pre-commit run –all-files` to do lint checks.

8. Generate documentation using `poetry run sphinx-build -b html docs/ docs/_build/html`.

9. Generate `requirements.txt` for automated testing using `poetry export –dev –without-hashes -f requirements.txt > requirements.txt`.

10. Commit the changes and raise a pull request.

Buy the developer a cup of coffee!

If you found the utility helpful you can buy me a cup of coffee using