S3 Downloader

API

s3 downloader to be used together with s3stalker from pytroll-collectors. S3downloader listens to messages from the s3stalker, and download files to configured destination. If the download is successful(ie. file exists on local disk a message is published to be used further downstream.

A yaml config file is needed like this: — logging:

log_rotation_days: 1 log_rotation_backup: 30 logging_mode: DEBUG

subscribe-topic:

• /yuhu

publish-topic: /idnt bucket: <name of the bucket> # Not needed, else used from the message uri download_destination: ‘./’ s3_kwargs:

anon: False profile: <profile name> # Optional client_kwargs:

endpoint_url: <url to the object store>

class trollmoves.s3downloader.FilePublisher(queue, nameservers)

A publisher for result files listening to a publish queue. Publishes the files via posttroll

run()

Run the publish loop getting messages from the publish queue.

stop()

Stops the file publisher.

class trollmoves.s3downloader.Listener(queue, config, subscribe_nameserver)

check_message(msg)

Check the message is valid type for this scope.

run()

“Run main loop subscribing to new messages adding these to a message queue.

stop()

Stops the file listener.

class trollmoves.s3downloader.S3Downloader(cmd_args)

Listen for posttroll messages and download files as described in the message from the configured object store (s3) bucket. Publish the successfuly downloaded file in as a posttroll message.

read_config()

Read the config file from cmd args.

setup_logging()

Init and setup logging.

start()

Start the threads for listening and publishing messages and handeling the download.

trollmoves.s3downloader.parse_args(args)

Parse the command line arguments.