python模块fabric踩坑记录

posted on | in | | read times:
words count: 18k | minutes to read ≈ 0:39

最近都在狂写脚本，好像变成 半个运维 一样…
刚好有需求写些部署工具，于是了解到了fabric这个模块。下面简单带过一下

什么是Fabric

引用fabric主页的介绍

Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return

意思就是Fabric是基于SSH的远程执行命令，并返回可调用的python对象的框架

Fabric2.x的版本与1.x相比，除了支持python3之外，还做了很多改动。网上很多博客都写的是1.x的版本，参考时要注意。
这里主要分析2.x的版本

安装什么的就不废话了，下面来用一下
参照他的示例:

1
2
3
4
5
6
7
 
>>> from fabric import Connection
>>> result = Connection('web1.example.com').run('uname -s')
>>> msg = "Ran {.command!r} on {.host}, got this stdout:\n{.stdout}"
>>> print(msg.format(result))

Ran "uname -s" on web1.example.com, got this stdout:
Linux

一目了然

下面在公司服务器上测试一下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
# 服务器ip: 10.10.22.13
# 用户: root

>>> from fabric import Connection
>>> result = Connection('10.10.22.13', user='root').run('uname -s')
>>> msg = "Ran {.command!r} on {.host}, got this stdout:\n{.stdout}"
>>> print(msg.format(result))

Traceback (most recent call last):
  File "/Users/yang/workspace/PycharmProjects/FP-project/inventory_allocate/test/fabric_test.py", line 10, in <module>
    result = Connection("10.10.22.13", user='root').run("uname -s")
  File "<decorator-gen-3>", line 2, in run
  File "/Users/yang/anaconda3/lib/python3.6/site-packages/fabric/connection.py", line 29, in opens
    self.open()
  File "/Users/yang/anaconda3/lib/python3.6/site-packages/fabric/connection.py", line 501, in open
    self.client.connect(**kwargs)
  File "/Users/yang/anaconda3/lib/python3.6/site-packages/paramiko/client.py", line 424, in connect
    passphrase,
  File "/Users/yang/anaconda3/lib/python3.6/site-packages/paramiko/client.py", line 715, in _auth
    raise SSHException('No authentication methods available')
paramiko.ssh_exception.SSHException: No authentication methods available

呵呵，我他妈就知道，代码里一行ssh的参数毛都没见到，这么容易连上就有鬼了（微笑
没的说，填坑要紧

源码分析

分析报错

1
2
3
4
5
6
7
8
9
10
11
12
13
 
Traceback (most recent call last):
  File "/Users/yang/workspace/PycharmProjects/FP-project/inventory_allocate/test/fabric_test.py", line 10, in <module>
    result = Connection("10.10.22.13", user='root').run("uname -s")
  File "<decorator-gen-3>", line 2, in run
  File "/Users/yang/anaconda3/lib/python3.6/site-packages/fabric/connection.py", line 29, in opens
    self.open()
  File "/Users/yang/anaconda3/lib/python3.6/site-packages/fabric/connection.py", line 501, in open
    self.client.connect(**kwargs)
  File "/Users/yang/anaconda3/lib/python3.6/site-packages/paramiko/client.py", line 424, in connect
    passphrase,
  File "/Users/yang/anaconda3/lib/python3.6/site-packages/paramiko/client.py", line 715, in _auth
    raise SSHException('No authentication methods available')
paramiko.ssh_exception.SSHException: No authentication methods available

我们可以看到调用堆栈上的错误回溯，定位到line 501，在实例化Connection对象后调用client.connect(**kwargs)时抽了…

直接摸过去，从Connection类出发往他祖宗上刨，下面先给出继承关系和Connection的关键部分，然后逐块拆分说明:

connection.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
 
class Connection(Context):
    host = None
    original_host = None
    user = None
    port = None
    ssh_config = None
    gateway = None
    forward_agent = None
    connect_timeout = None
    connect_kwargs = None
    client = None
    transport = None
    _sftp = None
    _agent_handler = None

    def __init__(
        self,
        host,
        user=None,
        port=None,
        config=None,
        gateway=None,
        forward_agent=None,
        connect_timeout=None,
        connect_kwargs=None,
    ):
        # config
        super(Connection, self).__init__(config=config)
        if config is None:
            config = Config()
        elif not isinstance(config, Config):
            config = config.clone(into=Config)
        self._set(_config=config)

        # host相关
        shorthand = self.derive_shorthand(host)
        host = shorthand["host"]
        err = (
            "You supplied the {} via both shorthand and kwarg! Please pick one."  # noqa
        )
        if shorthand["user"] is not None:
            if user is not None:
                raise ValueError(err.format("user"))
            user = shorthand["user"]
        if shorthand["port"] is not None:
            if port is not None:
                raise ValueError(err.format("port"))
            port = shorthand["port"]

        # ssh_config
        self.ssh_config = self.config.base_ssh_config.lookup(host)

        # original_host
        self.original_host = host

        # host
        self.host = host
        if "hostname" in self.ssh_config:
            self.host = self.ssh_config["hostname"]

        # user
        self.user = user or self.ssh_config.get("user", self.config.user)

        # port
        self.port = port or int(self.ssh_config.get("port", self.config.port))

        if gateway is None:
            if "proxyjump" in self.ssh_config:
                hops = reversed(self.ssh_config["proxyjump"].split(","))
                prev_gw = None
                for hop in hops:
                    if prev_gw is None:
                        cxn = Connection(hop)
                    else:
                        cxn = Connection(hop, gateway=prev_gw)
                    prev_gw = cxn
                gateway = prev_gw
            elif "proxycommand" in self.ssh_config:
                gateway = self.ssh_config["proxycommand"]
            else:
                gateway = self.config.gateway
        self.gateway = gateway

        # forward_agent
        if forward_agent is None:
            forward_agent = self.config.forward_agent
            if "forwardagent" in self.ssh_config:
                map_ = {"yes": True, "no": False}
                forward_agent = map_[self.ssh_config["forwardagent"]]
        self.forward_agent = forward_agent

        # connect_timeout
        if connect_timeout is None:
            connect_timeout = self.ssh_config.get(
                "connecttimeout", self.config.timeouts.connect
            )
        if connect_timeout is not None:
            connect_timeout = int(connect_timeout)
        self.connect_timeout = connect_timeout

        # connect_kwargs
        self.connect_kwargs = self.resolve_connect_kwargs(connect_kwargs)

        # client
        client = SSHClient()
        client.set_missing_host_key_policy(AutoAddPolicy())
        self.client = client

        # transport
        self.transport = None

重要的成员变量

class Connection
1
2
3
4
5
6
7
8
9
 
host = None             # 主机名或IP地址: www.host.com, 66.66.66.66
original_host = None    # 同host
user = None             # 系统用户名: root, someone
port = None             # 端口号（远程执行某些应用需提供）
gateway = None          # 网关
forward_agent = None    # 代理
connect_timeout = None  # 超时时间
connect_kwargs = None   # 连接参数（记住这个，非常重要）
client = None           # 客户端

构造函数参数

Connection.__init__() 
1
2
3
4
5
6
7
8
 
host
user=None
port=None
config=None
gateway=None
forward_agent=None
connect_timeout=None
connect_kwargs=None

这些就是我们在实例化Connection对象时可以控制的一些部分，比较重要的有config和connection_kwargs

构造函数主体

config

Connection.__init__() 
1
2
3
4
5
6
 
super(Connection, self).__init__(config=config)
if config is None:
    config = Config()
elif not isinstance(config, Config):
    config = config.clone(into=Config)
self._set(_config=config)

config成员变量是一个Config对象，它是调用父类Context.__init__()方法来初始化的。Context.__init__()定义如下:

class Context
1
2
3
4
5
6
7
8
9
10
 
class Context(DataProxy):
    def __init__(self, config=None):
        config = config if config is not None else Config()
        self._set(_config=config)

        command_prefixes = list()
        self._set(command_prefixes=command_prefixes)

        command_cwds = list()
        self._set(command_cwds=command_cwds)

具体过程是Context.__init__()初始化时调用_set()绑定了Config成员对象_config:

class DataProxy
1
2
3
4
5
 
def _set(self, *args, **kwargs):
    if args:
        object.__setattr__(self, *args)
    for key, value in six.iteritems(kwargs):
        object.__setattr__(self, key, value)

再通过加了@property的config()函数，使得connection对象能直接用self.config来引用_config:

class DataProxy
1
2
3
4
5
6
7
 
@property
def config(self):
    return self._config

@config.setter
def config(self, value):
    self._set(_config=value)

host, user, port

Connection.__init__() 
1
2
3
4
5
6
7
8
9
10
11
12
13
 
shorthand = self.derive_shorthand(host)
host = shorthand["host"]
err = (
    "You supplied the {} via both shorthand and kwarg! Please pick one."  # noqa
)
if shorthand["user"] is not None:
    if user is not None:
        raise ValueError(err.format("user"))
    user = shorthand["user"]
if shorthand["port"] is not None:
    if port is not None:
        raise ValueError(err.format("port"))
    port = shorthand["port"]

这段是处理host参数的。host可以有下面集中传入形式:

1
2
3
4
 
user@host:port  # 例如: root@10.10.10.10:6666
user@host       # 例如: root@10.10.10.10
host:port       # 例如: 10.10.10.10:6666
host            # 例如: 10.10.10.10

前三种会调用self.derive_shorthand(host)分别解析出self.host，self.user和self.port，最后一种需单独传入user，port。
如果用前三种传入方式的话，记得不要再重复传入user或port了，会抛出异常

源码是真他妈长啊，注释就占了一两百行 (微笑

ok，初始化的参数我们先到这里，因为它给的示例也就只有个host而已，一会再讲config_kwags和深入下config。我们现在先去报错的地方:

class Connection
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
kwargs = dict(
    self.connect_kwargs,
    username=self.user,
    hostname=self.host,
    port=self.port,
)
if self.gateway:
    kwargs["sock"] = self.open_gateway()
if self.connect_timeout:
    kwargs["timeout"] = self.connect_timeout
# Strip out empty defaults for less noisy debugging
if "key_filename" in kwargs and not kwargs["key_filename"]:
    del kwargs["key_filename"]
# Actually connect!
self.client.connect(**kwargs)  # 就是你了

看到最后一行，传参时将字典kwargs传了过去，kwargs里除了usr，host，port之外，还有一个connect_kwargs。我们看看client.connect()的定义:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 
def connect(
        self,
        hostname,
        port=SSH_PORT,
        username=None,
        password=None,      # 你
        pkey=None,          # 你
        key_filename=None,  # 还有你
        timeout=None,
        allow_agent=True,
        look_for_keys=True,
        compress=False,
        sock=None,
        gss_auth=False,
        gss_kex=False,
        gss_deleg_creds=True,
        gss_host=None,
        banner_timeout=None,
        auth_timeout=None,
        gss_trust_dns=True,
        passphrase=None,
    )

看到6, 7, 8行没？that’s it.

接下来我们改写一下一开始的示例:

  • 使用password:

    1
    2
    3
    4
    5
    6
    		 
    >>> from fabric import Connection
    >>> # my_password为10.10.22.13的root用户密码
    >>> conn = Connection('10.10.22.13', user='root', connect_kwargs={'password': '${my_password}'})
    >>> conn.run("uname -s")

    Linux

  • 使用key_filename:

    1
    2
    3
    4
    5
    6
    7
    		 
    >>> from fabric import Connection
    >>> # 使用key_filename参数前提需将你的ssh公钥（.pub后缀）添加到远程服务器的.ssh/authorized_keys file里
    >>> # id_rsa为私钥
    >>> conn = Connection('10.10.22.13', user='root', connect_kwargs={'key_filename': '${path to local .ssh dir}/${your id_rsa file}'})
    >>> conn.run("uname -s")

    Linux

朋友们，让我们举杯庆祝一下吧

connect_kwargs

我们趁此机会窥视一下connect_keargs的相关部分

Connection.__init__() 
1
 
self.connect_kwargs = self.resolve_connect_kwargs(connect_kwargs)

我们看到，当connect_kwargs为None时，会通过config成员变量动态增加属性connect_kwargs属性:

Connection.resolve_connect_kwargs() 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
def resolve_connect_kwargs(self, connect_kwargs):
    if connect_kwargs is None:
        connect_kwargs = self.config.connect_kwargs  # 调用__getattr__()动态增加属性
    elif "key_filename" in self.config.connect_kwargs:
        kwarg_val = connect_kwargs.get("key_filename", [])
        conf_val = self.config.connect_kwargs["key_filename"]
        connect_kwargs["key_filename"] = conf_val + kwarg_val

    if "identityfile" in self.ssh_config:
        connect_kwargs.setdefault("key_filename", [])
        connect_kwargs["key_filename"].extend(
            self.ssh_config["identityfile"]
        )

    return connect_kwargs

在__getattr__方法里又调用了类方法_get()将connect_kwargs传到key:

DataProxy.__getattr__() 
1
2
3
4
5
6
7
8
9
10
11
12
13
 
def __getattr__(self, key):
    try:
        return self._get(key)  # 调用_get()
    except KeyError:
        if key in self._proxies:
            return getattr(self._config, key)
        err = "No attribute or config key found for {!r}".format(key)
        attrs = [x for x in dir(self.__class__) if not x.startswith('_')]
        err += "\n\nValid keys: {!r}".format(
            sorted(list(self._config.keys()))
        )
        err += "\n\nValid real attributes: {!r}".format(attrs)
        raise AttributeError(err)

调用_get()后返回一个DataProxy对象value:

DataProxy._get()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 
def _get(self, key):
    if key in (
        '__setstate__',
    ):
        raise AttributeError(key)
    value = self._config[key]
    if isinstance(value, dict):
        keypath = (key,)
        if hasattr(self, '_keypath'):
            keypath = self._keypath + keypath
        root = getattr(self, '_root', self)
        value = DataProxy.from_data(
            data=value,
            root=root,
            keypath=keypath,
        )
    return value

是不是晕了？呵呵没关系，我他妈也是。就让他随风而去吧

fab命令

安装完fabric后会连同fab工具一起装到python/bin下，在终端输入fab -h查看命令参数
简单来说，fab干这么件事，直接执行当前目录下的fabfile.py脚本里的函数，下面介绍具体如何写fabfile.py

首先按照国际惯例，导入包:

1
2
 
from fabric import Connection
from invoke import task

实例化COnnection对象:

1
2
 
# ${}里的内容自行填充
conn = Connection("${remote host}", user='${remote user}', connect_kwargs={'password': "${remote user's password}"})

定义一个功能函数:

1
2
3
 
@task
def execute(c):
    conn.run("uname -s")

这个@task装饰器是必须加的，保证fab能直接执行，参数c不用管它，但不能定义成与你实例化的conn同名，原因后面会说

保存为fabfile.py:

fabfile.py
1
2
3
4
5
6
7
8
 
from fabric import Connection
from invoke import task

conn = Connection("${remote host}", user='${remote user}', connect_kwargs={'password': "${remote user's password}"})

@task
def execute(c):
    conn.run("uname -s")

在终端执行（服务器系统为Linux）:

1
2
 
$ fab execute
Linux  # Linux下的输出

说回参数c，c在这里实际上是本地连接，是localhost的一个Connection对象。我们可以试一下:

fabfile.py
1
2
3
4
5
6
 
from fabric import Connection
from invoke import task

@task
def execute(c):
    c.run("uname -s")

在终端执行（我的系统为Mac）:

1
2
 
$ fab execute
Darwin  # Mac下的输出

一目了然

到这里大家应该大致明白fab是干啥的了。这样的玩法就多了，下面举几个例子:

fabfile.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
from fabric import Connection
from invoke import task

conn = Connection("${remote host}", user='${remote user}', connect_kwargs={'password': "${remote user's password}"})

@task
def uname_local(c):
    c.run("uname -s")

# 列出某路径下的文件
@task
def ls_remote(c, dir_path):
    with conn.cd(dir_path):
        conn.run("ls -la")

# 还能在函数里实例化Connection对象
@task
def uname_rmt(c, host, user, password):
    con = Connection(host, user=user, connect_kwargs={'password': password})

    con.run("uname -s")

执行:

1
2
3
4
5
6
7
8
9
 
$ fab uname_local
Darwin

$ fab ls_remote /home
trendy
td_root

$ fab uname_rmt 10.10.22.13 root ******* # 这个密码就不放出来了
Linux

具体的Connection还封装了哪些命令，就需要大家在使用中探索了

fab 1.x 与 fab 2.x

fab 1.x与fab 2.x最大的不同就是fab 2.x没有了env模块，所有的操作都基于Connection对象完成，对于fab命令的调用，将暴露方法封装到invoke模块，使其独立出来。这是我觉得最大的不同。只是现在还没有很多的博客写到fab 2.x的一些特性，官方文档也很简单，还是需要大家花时间阅读下源码才能清楚里面的逻辑

有时间的话会再对fabric做个详细剖析

---------------------------------END---------------------------------
手抖一下？