Browse Source

Add parallelization to `tootctl search deploy` (#12051)

* Add parallel gem

* Modify parallel option in tootctl search deploy

* Add paralell option to tootctl search deploy

* Change 1 to false

* Clean up

* Rename --parallel to --processes
tags/v3.0.0rc3
Takeshi Umeda 2 weeks ago
parent
commit
0ce0baa9b5
3 changed files with 23 additions and 2 deletions
  1. 1
    0
      Gemfile
  2. 1
    0
      Gemfile.lock
  3. 21
    2
      lib/mastodon/search_cli.rb

+ 1
- 0
Gemfile View File

@@ -67,6 +67,7 @@ gem 'oj', '~> 3.9'
gem 'ostatus2', '~> 2.0'
gem 'ox', '~> 2.11'
gem 'parslet'
gem 'parallel', '~> 1.17'
gem 'posix-spawn', git: 'https://github.com/rtomayko/posix-spawn', ref: '58465d2e213991f8afb13b984854a49fcdcc980c'
gem 'pundit', '~> 2.1'
gem 'premailer-rails'

+ 1
- 0
Gemfile.lock View File

@@ -748,6 +748,7 @@ DEPENDENCIES
ox (~> 2.11)
paperclip (~> 6.0)
paperclip-av-transcoder (~> 0.6)
parallel (~> 1.17)
parallel_tests (~> 2.29)
parslet
pg (~> 1.1)

+ 21
- 2
lib/mastodon/search_cli.rb View File

@@ -6,6 +6,7 @@ require_relative 'cli_helper'

module Mastodon
class SearchCLI < Thor
option :processes, default: 2, aliases: [:p]
desc 'deploy', 'Create or update an ElasticSearch index and populate it'
long_desc <<~LONG_DESC
If ElasticSearch is empty, this command will create the necessary indices
@@ -13,10 +14,28 @@ module Mastodon

This command will also upgrade indices if the underlying schema has been
changed since the last run.

With the --processes option, parallelize execution of the command. The
default is 2. If "auto" is specified, the number is automatically
derived from available CPUs.
LONG_DESC
def deploy
processed = Chewy::RakeHelper.upgrade
Chewy::RakeHelper.sync(except: processed)
processed = Chewy::RakeHelper.upgrade(parallel: processes)
Chewy::RakeHelper.sync(except: processed, parallel: processes)
end

private

def processes
return true if options[:processes] == 'auto'

num = options[:processes].to_i

if num < 2
nil
else
num
end
end
end
end

Loading…
Cancel
Save