Changelog#
0.19.0#
February 22, 2024
#332
- Add Context
0.18.0#
February 15, 2024
#339
- Azure Blob Filesystem#338
- Fix is_valid_handle to work for unsaved handles.#337
- Allow arguments in a PartialTask call to override previously specified arguments#335
- Explicit top level redun.* exports#333
- Surface job_def_extra option
0.17.0#
November 03, 2023
#327
- Utility functions to support experimental k8s work.#328
- Fix CI failures in main#326
- Add documentation of the tag system#324
- Add support for argv lists to script#323
- Add execution_id to launch script for tagging purposes#322
- Handling race conditions when caching values#321
- Move gcsfs to optional google extra dependencies
0.16.2#
August 25, 2023
#319
- avoid Screen.query method#317
- Add DockerExecutor volumes config#315
- Efficiently compute Exec and Job status from result type#314
- add missing repl hotkey for search screen#313
- Remove duplicate federated tasks
0.16.1#
July 27, 2023
#311
- update MANIFEST.in for console/style.css#310
- skip test since shutdown doesnt actually run in 3.8#309
- Test task and scheduler level catch cache busting#307
- Add Console docs
0.16.0#
June 21, 2023
#303
- Fix boto thread safe#300
- Console extras#301
- Fix scrapping typo#299
- Add batch job tags at subission time#297
- broaden the scope of console executions –find to include args#294
- Add TUI Console to redun cli#295
- Update stale example output#293
- Example of running conda in a docker container#292
- Allow setting no role for aws batch jobs#291
- Removed pinned boto version for doc requirements#73
- split out version into it’s own module#68
- GCP batch support v2#72
- Fix link on the “Scheduling redun workflows” docs pageuse legacy solver for docs publishing
fix sphinx docs building
#71
- Update design.md
0.15.0#
May 01, 2023
#286
- Uncap botocore to allow better dependency solving#285
- DE-8667 ensure to call record_job_start even when executor is not defiend#275
- Drop python 3.6#284
- DE-8613 make Handle.is_valid() more graceful to missing recordings#283
- Add lambda_federated_task scheduler task with tests#282
- small fix for graph reduction docs#281
- Update TaskExpression names to match code snippet#280
- [DE-8465] Fair Share Scheduling support.#278
- Release 0.14.0#261
- Provide integration code for redun and a proxy#277
- Add task_def option to federated_task config#247
- Publish federated tasks#270
- DE-8068 Pass through missing aws batch options
0.14.0#
March 13, 2023
This release primarily introduces “federated tasks”, a mechanism for working with tasks where you do not have to import the implementation code.
#261
- Provide integration code for redun and a proxy#277
- Add task_def option to federated_task config#247
- Publish federated tasks#270
- DE-8068 Pass through missing aws batch options
0.13.0#
February 27, 2023
#272
- DE-7556: Update job definition sanitizing to be recursive#182
- Check CSE even for completed jobs#269
- Bump SQLAlchemy to 2.0, use future=True flag#268
- Follow-up on SQLAlchemy 2.0 compatibility#266
- DE-7776 Upgrade sqlalchemy to 1.4 and fix deprecation warnings#22
- Add k8s executor#253
- Try to clarify and document Value and Handle#63
- remove develop mode for installing redun during docs build#254
- DE-7282 Simplify job arguments and JobArrayer#255
- Clarify role arn and fix for sphinx 6.0#61
- Pass AWS_DEFAULT_REGION env variable to container#244
- DE-6763 Add IFile and ContentFile#60
- Fix import statements with lowercase config
0.12.0#
November 29, 2022
#241
- Remove length restriction on redun type names#242
- Use previews for value with no loaded class#243
- DE-6625 Catch and update eval_hash on race condition#240
- Specify user and password in example readme#232
- DE-6585 Raise a more informative error message for File errors
0.11.1#
November 01, 2022
#234
- Temporarily cap botocore version due to upstream bug
0.11.0#
November 01, 2022
#227
- Add –wait option to redun launch#221
- Add launch to run to allow remote runs#207
- DE-6252 Upgrade mypy and remove python pin#217
- DE-6397 Add pre commit for all linting/formatting
0.10.1#
October 20, 2022
#219
- Fix nested value iteration for dataclasses instantiated from subscripted generics
0.10.0#
October 17, 2022
#218
- Add alias executors#216
- DE-6353 Add env var parsing to config files#215
- Update package version to 0.9.1, skip 0.9.0 in changelog#213
- Backport fixes
0.9.1#
September 30, 2022
#206
- Add missing get_hash to ShardedS3Dataset#203
- Update docs to match Expression repr changes#205
- Temporarily pin Python 3.10.6 to avoid mypy bug on 3.10.7#195
- Smarter job def#173
- Run tox in parallel(up to CPU count)#202
- Remove PR buildspec, codebuild was made redundant by GH actions#201
- DE-6221 - Make GH actions match codebuild#200
- fix-handle-arg-serialization#199
- implement an easier to read repr for Expressions#198
- Fix handling of multi-node job statuses and logs#197
- fix bug with mark_dup staging
0.8.16#
September 08, 2022
#194
- Increase defaultulimits
fornofile
(number of open file descriptors) for multi-node aws batch jobs#193
- handle dataclass types correctly#189
- Support nested iteration and mapping for dataclasses#192
- A few small bug fixes#191
- DE-6030 Fix expression cycle detection for cached expressions#188
- DE-5961 Allow functions decoratored by @task and @scheduler_task to processed by autodoc
0.8.15#
August 18, 2022
#179
- Add job stitching to subrun#185
- include api reference in sphinx docs#181
- DE-5801 Add interactive config for docker executor#180
- [DE-5731] Make vizualization functionality optional when pygraphviz is unavailable#178
- Update docs to describe db migrations#175
- DE-5591 Fix Common Subexpression Elimination (CSE)#47
- pass self.region to submit_command() from AWSBatchExecutor._submit_single_job()#174
- Update test_pull_request.yml#160
- Call graph visualizer#126
- Improved display of job status#125
- Better support for previewing large values#170
- add support for parsing datetime#38
- Bump pyspark from 3.1.1 to 3.1.3#37
- working on fixing docs build
0.8.14#
July 5, 2022
#165
- Switch conda package to noarch#168
- Bump version ofblack
inredun_server
0.8.13#
June 16, 2022
#162
- DE-5345 Fx multi-node doing caching#155
- Update linting tools#161
- DE-5312 Expand the try-catch to include input parsing#157
- [DE-3475] Fix UDF type arg being ignored, add some Spark tests#156
- De4653 hash code alt#149
- Generalize file staging#159
- Fix aws batch shared memory#152
- Added docstrings to key Glue job functions
0.8.12#
June 01, 2022
#153
- DE-5198 Don’t drop containerProperties when sanitizing job defs#151
- remove unused and broken extract_tar import
0.8.11#
May 29, 2022
#30
- Add missing data.tsv and fix cleanup default
0.8.10#
May 26, 2022
#146
- Update moto to 3.1.10#145
- DE-5109 Fix missing--array-job
argument forredun oneshot
#144
- Add more docs to docker executor functions/classes#137
- DE-4915 Breakout DockerExecutor and consolidate scratch and code packaging
0.8.9#
May 17, 2022
#131
- [DE-4809] Use AWS job run insights to get better tracebacks for glue jobsMerge remote-tracking branch ‘origin/main’
#138
- resolve conflicts with public and private main#33
- Fixed typos and minor grammatical stuff like missing commas, etc. in the documentation and README.md#32
- Update README.md#136
- Ignore flake8 F401 errors in__init__.py
files
0.8.8#
April 20, 2022
#129
- AWS Batch multi-node executor#130
- Update README.md because compositiblity is not a word.#128
- DE-4761 fix script reuniting in aws batch executor#127
- Keep ‘redun log’ stable tag sorting even for complex tag values#124
- DE-4689 File encoding v2#119
- DE-4636 Fix for S3FileSystem threading issues#120
- Fix job clear bug#80
- DE-4600 Working example of catch_all#116
- Bump version to 0.8.7
0.8.7#
March 11, 2022
#115
- DE-4596 Handle case where containerProperties is missing#113
- simplify apply_tags task#111
- Include restored traceback when restoring from pickled data#112
- Add apply_job_tags and apply_execution_tags
0.8.6#
February 16, 2022
#103
-make setup
should be runnable more than once in Docker examples#102
- Minor usability improvements to ShardedS3Dataset#93
- Implement subrun()#100
- Suggestion for shared database usage#97
- Add lint job to github actions. Upgrade black. Ignore F811.#92
- Streamline local executor exec methods#95
- Implement scheduler.get_job_status_report()#94
- Implement Config.get_config_dict()#96
- Advertizemax_value_size
andvalue_store_path
when values are too large for db#90
- Better error mesg if task is missing in registry#91
- Fix import order and line spacingMerge pull request #87 from insitro/DE-4210_db_aws_region_from_env
#87
- Use AWS_REGION env variable for DB secret lookup
0.8.5#
January 11, 2022
#66
- Treat functions as a valid redun ValueMerge pull request #65 from insitro/DE4020-shardeddataset-save
Remove superfluous type hint
#70
- improve module not found error#76
- Link docs#77
- Use a separate boto session per thread#74
- Fix aws batch test warnings: DescribeJobs operation: The security token included in the request is invalid#75
- DE-4104 Proposed fix for the s3fs dep issues#54
- DE-3969 Evaluate default args in case they have expressions#73
- DE-4102 Add inhert_cache=True to Column to suppress sqlalchemy warningMerge pull request #72 from insitro/typos_and_cleanups
Correct misspellings of executor
Remove superfluous parens
Merge pull request #71 from insitro/add_build_timeout
Add gh-actions build timeout
#67
- Generate public documentsAdd SQL query helper function for glue
Add from_data constructor for ShardedS3Dataset
#13
- no need to use editable mode for redun install in examplesMerge pull request #62 from insitro/DE-3993-override-source
Clarifying comments on task.source during unpickling
0.8.4#
December 02, 2021
Merge remote-tracking branch ‘pub/main’
Merge pull request #58 from insitro/DE-3970-empty-string-vs-none-namespace
Merge pull request #45 from insitro/DE3937-glue-imports
#11
- Fixes for several problems davidek saw#55
- DE-3967 Fix case where subpromise is resolved first for Promise.all()#46
- DE-3940 Variadic arguments should contribute to the eval_hash#48
- Show a better error for workflow script not found#43
- DE-3900 Handle missing logstream#5
- Fix typos discovered by codespell
0.8.3#
November 10, 2021
#32
- Fix version cli arg conflict#38
- Add python 3.10 to codebuild#36
- Clarify release triggering documentationMerge pull request #33 from insitro/prevent-double-ci-runs - Prevent double github actions runs
0.8.2#
November 10, 2021
#23
- Add support for task wrappers#4
- GitHub Action to run tox on every pull request#3
- Import Executor for lines 36 and 38#28
- Add more influence text for bioinfo and ML frameworks#26
- Separate the various concepts of options#25
- Document development installation and testing#27
- DE-3811 Allow naked task decoratorMerge pull request #21 from insitro/glue_py_bugfix
Add clarifying comments
Correctly format extra_py_files arg
0.8.1#
October 24, 2021
#11
- improve job and promise memory usage
0.8.0#
October 13, 2021
#7
- Try relaxing s3fs version range#6
- DE-3650 set Task.source when deserializing#5
- DE-3641 Add testing example#4
- [DE-3531] Make additional python files work for glue jobs#3
- Use public docker images#2
- Update postgres install instructions#1
- Add testing buildspec.Beginning of public repo
#312
- Update html links to markdown#28
- Example of making postgres optional#305
- Compute hash of ShardedS3Dataset after it’s returned#304
- Remove unnecessary DO NOT USE warning that contained core references#300
- adding insitro redun.ini examples#303
- DE-3485 clean up design doc#301
- Fixes from library cleanup#298
- DE-3490 Clean examples#299
- Small lib cleanups#297
- Remove unused variables in AWS glue code#293
- add scripts for making a public version of the redun repo
0.7.4#
September 30, 2021
#294
- More redun examples#292
- Add git commit and origin_url tags to execution#291
- Add more debugging logs for executors [WIP]
0.7.3#
September 21, 2021
#286
- Revert “Update latest setuptools-conda with dependency version fix “#288
- Move AWS batch statuses to constants
0.7.2#
September 20, 2021
#286
- Update latest setuptools-conda with dependency version fix#285
- DE-3421 Add executor-based job tags#250
- DE-3433 Add--rerun
option toredun run
0.7.1#
September 16, 2021
#282
- DE-3399 Add tag APIs for redun_server#272
- DE-2651 Add Spark executor#277
- DE-2483 value store#278
- Update postgres example config#283
- Ignore self and cls variables for docstring test
0.7.0#
September 09, 2021
#209
- DE-1592 Tag system v2#253
- DE-2945 Use ProcessPoolExecutor through a separate executor#280
- DE-3342 refactor CallGraphQueryMerge branch ‘master’ of github.com:insitro/redun
update journal
#276
- small tutorial fixes#274
- Test for numpy docstring adherence#273
- Fix minor typos in docs#271
- Update redun server’s ecs cluster & setup CD for redun-dev#260
- DE-3094 Implement a fast Files search page#270
- add sys.exit(1)Merge branch ‘master’ of github.com:insitro/redun
update journal
#268
- remove recursive query for job executions#267
- fix migration with a missing commit
0.6.1#
August 04, 2021
#265
- avoid Session as context use#264
- Abandon release if failures are encountered during install or build
0.6.0#
August 02, 2021
#261
- DE-3105 Add Job.execution_id migration#259
- DE-3091 Add indexes to commonly queries columns#258
- Small improvements to File such as file size and staging/copy defaults
0.5.1#
July 16, 2021
#256
- Quote the DB password when creating DB URIs#255
- Add support for Python 3.9#245
- DE-2923 Extending the tutorial#252
- DE-2939 Improve sort of sections in dataflow#244
- Allow for non-python function as Task.source in dataflow visualization#251
- DE-2922 Small fixes and improvements to batch executor and File#249
- Update redun server to handle new task serialization format#248
- DE-2900 Show db too new message
0.5.0#
June 28, 2021
#246
- tee stdout and stderr#229
- Backfill lonely Tasks, and update Task/Value serialization#241
- DE-2001 Add File support for http, https, and ftp#240
- DE-2850 Guided tutorial through workflow examples#242
- add missing use_tempdir#238
- Remove rogue . in alembic version info#237
- use python3 in Makefile
0.4.15#
June 15, 2021
#235
- fix: bump boto3 floor version for required botocore functionality#232
- [DE-2761] Make tee tolerant of write errors for script batch jobs#233
- DE-2632 – Handle case where non-redun jobs have matching prefix#234
- DE-2711 – Fix optional cli args#228
- Every time a db.Task is recorded, also record it as a db.Value
0.4.14#
June 07, 2021
#224
- DE-2713 Add batch_tags option#230
- Fix job status#222
- upstream useful changes from tags-flag#219
- DE-2660 Use a default root task for non-TaskExpressions
0.4.13#
May 18, 2021
#220
- DE-2637 fix hashing of task_options_update#204
- DE-2619 Use O(1) queries to speedup record serialization#218
- DE-2635 Show unknown CallNodes for unfinished jobs#217
- show keyword arguments#216
- Fix isort line length#215
- DE-2623 Dont use recursive for getting execution jobs#213
- fix path term parsing#212
- fix: redun server ECS service name in merge spec#208
- Scope redun_server DB sessions at the request level#210
- Cleanup logging of migrations#211
- DE-2599 Use wait_until in aws batch tests to fix flaky tests
0.4.12#
May 07, 2021
#206
- Add method to clone RedunBackendDB with connection pool sharing#196
- DE-2325 Add database versioning commands#201
- Add quick script to generate release notes
0.4.11#
April 22th, 2021
#198
- Add support for configuration only task args#197
- [DE-2428] Fix typed list check#192
- DE-2434 Add more common tasks to functools#194
- decouple scheduler from oneshot#186
- Dockerize redun server, update directory layout and utils, add specs for prod deployment#190
- DE-2464 Add postmortem debugging
0.4.10#
April 12th, 2021
#188
- Don’t let docker change terminal to raw mode#187
- Tasks should allow novel kwargs#180
- Use amazonlinux default pythons#185
- Support job timeouts on batch#182
- Lazy operators for redun Expressions
0.4.9#
March 23rd, 2021
#183
- add py.typed#177
- Support list args from cli#178
- Fix settrace monkeypatch to restore debugging ability#179
- DE-2370 Give array jobs a unique uuid#181
- sqlalchemy 1.4.0 no longer allows postgres:// gotta be postgresql://#176
- Improve pickle preview for constructor and new#173
- Allow pycharm’s debugger to work with redun#175
- Set choices on parser for enum args#174
- Allow use of id prefixes with push/pull commands#171
- Make S3 repositories work#172
- Match python 3.7 and 3.8 micro versions to match codebuild image
0.4.8#
March 10th, 2021
#111
- Add concept of remote repos#169
- Remove invalid positional arg in get_or_create_job_definition call#147
- Dir should have File as subvalues for better dataflow recording#165
- Fix lack of caching for catch expressions#164
- Fix PartialTask’s options() and partial() calls so that they interact correctly#163
- Imports executors in the init#155
- Use config_dir with redun_server
0.4.7#
February 24th, 2021
WARNING: This version contains a bug in the get_or_create_job_defintion
call in batch_submit
. Do not use this version.
#156
- Automatic publishing of packages and docs#153
- Use existing job def#116
- Display dataflow#154
- Fix data provenance recording for seq scheduler task#152
- Fix pickling expression upstreams#136
- Add redux to redun_server#151
- Record stderr from scripts on batch#149
- Add support for generating DB URI from AWS secret#150
- Document max value size#146
- Cryptic error for large falues#148
- Simplify Scheduler.run() to take expressions#145
- Add nout task option for tuples#144
- Increase sqlalchemy requirement to 1.3.17#143
- Package on submit not start
0.4.6#
February 3rd, 2021
#141
- Only gather inflight jobs on batch on first submission
0.4.5#
January 28th, 2021
#139
- Propagate batch script errors#137
- Override CannotInspectContainerError batch errors#138
- Fix pickle preview for classes where the module can’t be found#133
- Small fixes from demo talk#132
- Small improvements to File.copy_to and self-stagin
0.4.4#
January 15th, 2021
#131
- Fix catch dataflow#134
- Add notebook example of redun scheduler evaluation#128
- Make redun compatible with sqlalchemy-1.4.0b1#129
- Add pickle_preview for unknown classes#130
- Fix catch dataflow#127
- Add FAQ page to docs#126
- Require sorted imports
0.4.3#
January 5th, 2021
#122
- Stronger type checking for task calls#101
- Record CallNodes when an exception is raised#86
- Scheduler tasks
0.4.2#
January 4th, 2021
#121
- Array job reuniting fix
0.4.1#
December 23rd, 2020
#119
- Bugfix to correctly restart job array monitor thread
0.4.0#
December 15th, 2020
#83
- Detect and submit job arrays to AWS batch#114
- Adds job definition option to run container in privileged mode
0.3.12#
December 10th, 2020
#76
- Improve querying of logs
0.3.11#
December 8th, 2020
#109
- Permalink update in README#108
- Automated release
0.3.10#
December 3rd, 2020
#104
- use ECR for postgres image#95
- Hard fail on script errors#100
- Show more information in logs and traceback#102
- Fix check-valid=shallow to use the original call node#98
- Skip license check when building conda packages#105
- Typecheck map_nested_value#103
- Fix script reactivity to inputs and outputs#106
- Small clean up of batch logs
0.3.9#
November 25th, 2020
#96
- Default to interactive debugging#81
- Allow REDUN_CONFIG environment variable to specify config directory#92
- DE-1922 tolerate missing logs for failed jobs
0.3.8#
November 18th, 2020
#89
- Respect no-cache for job reuniting.#88
- Assume batch output after completion is valid.#87
- Fix filesystem caching and Dir hashing caching.#85
- Add step to publish pypi package in publish script.#84
- Fix package name in dependencies notes in README.
0.3.7#
November 12th, 2020
#80
- redun import paths should take precedence over system imports.#79
- fix default arg parsing and prefix args.
0.3.6#
November 10th, 2020
#73
- Allow users to customizesetup_scheduler()
.
0.3.5#
November 10, 2020
#77
- Check version of redun cli in docker container.
0.3.4#
October 29th, 2020
#72
- Use current working directory when importing a module.#64
- Some optimizations for AWS Batch large fanout.
0.3.3#
October 28th, 2020
#71
- Don’t fetch batch logs when debug=True
0.3.2#
October 27th, 2020
#66
- Fix import_script to properly support module-style
0.3.1#
Fix bug with using s3fs >= 0.5
0.3#
October 20th, 2020
Improve display of errors and logs for AWS Batch jobs.
0.2.5#
October 14th, 2020
#57
- Improve redun traceback for failed jobs.#56
- Fix local shell error propagation.#54
- Add documentation on required dependencies.
0.2.4#
October 6, 2020
Encourage defining task namespaces by raising a warning. The warning can be ignored using a configuration option.
0.2.3#
September 25, 2020
Fixes FileNotFoundError occurring when using AWS Batch tasks, by avoiding the s3fs cache.
0.2.2#
August 27, 2020
Require database credentials to be specified by environment variables
0.2.1#
August 9, 2020
Fix duplicate upstream bug.
0.2.0#
August 7, 2020
Add support for Python 3.8
0.1.1#
July 29, 2020
Drop dependency on bcode as it has no conda package and the repo appears abandoned.
0.1#
Initial release.