用户您好!请先登录!

Archives2019年11月11日

知识图谱在美团智能场景下的应用与实践

导读:目前为止 IT 产业经历了六次浪潮,分别为:大型机时代,小型机时代,个人电脑时代,桌面互联网时代,移动互联网时代和 AIOT 时代。在这些时代背后可以发现是人机交互方式的变化:从鼠键交互,到触控交互,再到语音智能交互,可以看到人机交互的方式在向更自然更直接化的方式演进。今天会和大家分享基于知识图谱的问答在美团智能交互场景中的应用和演进。

今天的介绍会围绕下面三点展开:

  • 智能交互背景介绍
  • 受限场景问答应用和演进
  • 复杂场景问答应用和演进

智能交互背景介绍

1. 智能交互的划分

知识图谱在美团智能场景下的应用于实践

智能交互的划分基本上是根据人类需求拆分:

  • 检索式交互—信息获取,比较经典的方法 FAQ:QA 匹配,QQ 匹配;
  • 任务式交互—执行任务,比如订机票 ( 酒店 ) 的特定任务;
  • 闲聊式交互—娱乐与休闲,基于深度学习的端到端的学习系统。

阅读更多

Google Container Tools: Skaffold

Skaffold is a command line tool that facilitates continuous development for Kubernetes applications. You can iterate on your application source code locally then deploy to local or remote Kubernetes clusters. Skaffold handles the workflow for building, pushing and deploying your application. It also provides building blocks and describe customizations for a CI/CD pipeline.

20个月测试,40次迭代,这款谷歌Kubernetes自动化开源工具通用了

Features

  • Fast local Kubernetes Development
    • optimized source-to-k8s – Skaffold detects changes in your source code and handles the pipeline to buildpush, and deploy your application automatically with policy based image tagging and highly optimized, fast local workflows
    • continuous feedback – Skaffold automatically manages logging and port-forwarding
  • Skaffold projects work everywhere
    • share with other developers – Skaffold is the easiest way to share your project with the world: git clone and skaffold run
    • context aware – use Skaffold profiles, user level config, environment variables and flags to describe differences in environments
    • CI/CD building blocks – use skaffold run end-to-end or just part of skaffold stages from build to deployment in your CI/CD system
  • skaffold.yaml – a single pluggable, declarative configuration for your project
    • skaffold init – Skaffold discovers your files and generates its own config file
    • multi-component apps – Skaffold supports applications consisting of multiple components
    • bring your own tools – Skaffold has a pluggable architecture to allow for different implementations of the stages
  • Lightweight
    • client-side only – Skaffold does not require maintaining a cluster-side component, so there is no overhead or maintenance burden to your cluster.
    • minimal pipeline – Skaffold provides an opinionated, minimal pipeline to keep things simple

Installing Skaffold

Stable binary

For the latest stable release download and place it in your PATH:

https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64

Run these commands to download and place the binary in your /usr/local/bin folder:

curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64
chmod +x skaffold
sudo mv skaffold /usr/local/bin

Latest bleeding edge binary

For the latest bleeding edge build, download and place it in your PATH:

https://storage.googleapis.com/skaffold/builds/latest/skaffold-linux-amd64

Run these commands to download and place the binary in your /usr/local/bin folder:

curl -Lo skaffold https://storage.googleapis.com/skaffold/builds/latest/skaffold-linux-amd64
chmod +x skaffold
sudo mv skaffold /usr/local/bin

 

A Glance at Skaffold Workflow and Architecture

Skaffold simplifies your development workflow by organizing common development stages into one simple command. Every time you run skaffold dev, the system

  1. Collects and watches your source code for changes
  2. Syncs files directly to pods if user marks them as syncable
  3. Builds artifacts from the source code
  4. Tests the built artifacts using container-structure-tests
  5. Tags the artifacts
  6. Pushes the artifacts
  7. Deploys the artifacts
  8. Monitors the deployed artifacts
  9. Cleans up deployed artifacts on exit (Ctrl+C)

What’s more, the pluggable architecture is central to Skaffold’s design, allowing you to use the tool you prefer in each stage. Also, Skaffold’s profiles feature grants you the freedom to switch tools as you see fit depending on the context.

For example, if you are coding on a local machine, you can configure Skaffold to build artifacts with local Docker daemon and deploy them to minikube using kubectl, the Kubernetes command-line interface and when you finalize your design, you can switch to the production profile and start building with Google Cloud Build and deploy with Helm.

Skaffold supports the following tools:

  • Build
    • Dockerfile locally
    • Dockerfile in-cluster (kaniko)
    • Dockerfile on cloud (Google Cloud Build)
    • Bazel locally
    • Jib Maven/Gradle locally
  • Test
    • with container-structure-test
  • Tag
    • tag by git commit
    • tag by current date&time
    • tag by environment variables based template
  • Push
    • don’t push – keep the image on the local daemon
    • push to registry
  • Deploy
    • Kubernetes Command-Line Interface (kubectl)
    • Helm
    • kustomize

architecture

Besides the above steps, skaffold also automatically manages the following utilities for you:

  • forwards container ports to your local machine using kubectl port-forward
  • aggregates all the logs from the deployed pods

Documentation

Documentation for latest release: https://skaffold.dev

Documentation for latest build: https://skaffold-latest.firebaseapp.com

 

饿了么推荐系统的0到1

随着移动互联网的发展,用户使用习惯日趋碎片化,如何让用户在有限的访问时间里找到想要的产品,成为了搜索/推荐系统演进的重要职责。作为外卖领域的独角兽, 饿了么拥有百万级的日活跃用户,如何利用数据挖掘/机器学习的方法挖掘潜在用户、增加用户粘性,已成为迫切需要解决的问题。

个性化推荐系统通过研究用户的兴趣偏好,进行个性化计算,发现用户的兴趣点,从而引导用户发现自己的信息需求。一个好的推荐系统不仅能为用户提供个性化的服务,还能和用户之间建立密切关系,让用户对推荐产生依赖。

本次分享介绍饿 了么如何从0到1构建一个可快速迭代的推荐系统,从产品形态出发,包括推荐模型与特征工程、日志处理与效果评估,以及更深层次的场景选择和意图识别

在携程个性化推荐与人工智能meetup上,已经就以上几部分做了整体上的说明,本文将就其中模型排序与特征计算的线上实现做具体说明,同时补充有关业务规则相关的洗牌逻辑说明,力图从细节上还原和展示饿了么美食推荐系统。

一、模型排序

1.设计流程

对于任何一个外部请求, 系统都会构建一个QueryInfo(查询请求), 同时从各种数据源提取UserInfo(用户信息)、ShopInfo(商户信息)、FoodInfo(食物信息)以及ABTest配置信息等, 然后调用Ranker排序。以下是排序的基本流程(如下图所示):

  • 调取RankerManager, 初始化排序器Ranker:
  • 根据ABTest配置信息, 构建排序器Ranker;
  • 调取ScorerManger, 指定所需打分器Scorer(可以多个); 同时, Scorer会从ModelManager获取对应Model, 并校验;
  • 调取FeatureManager, 指定及校验Scorer所需特征Features。
  • 调取InstanceBuilder, 汇总所有打分器Scorer的特征, 计算对应排序项EntityInfo(餐厅/食物)排序所需特征Features;
  • 对EntityInfo进行打分, 并按需对Records进行排序。
饿了么推荐系统:从0到1(天天外卖,你可不知道的程序里面原理)

阅读更多

HBase架构简析

物理上,Hbase 是由三种类型的 server 组成的的主从式(master-slave)架构:

  • Region Server,负责处理数据的读写请求,客户端请求数据时直接和 Region Server 交互。
  • HBase Master,负责 Region 的分配,DDL(创建,删除 table)等操作。
  • Zookeeper,作为 HDFS 的一部分,负责维护集群状态。

当然底层的存储都是基于 Hadoop HDFS 的:

  • Hadoop DataNode 负责存储 Region Server 所管理的数据。所有的 HBase 数据都存储在 HDFS 文件中。Region Server 和 HDFS DataNode 往往是分布在一起的,这样 Region Server 就能够实现数据本地化(data locality,即将数据放在离需要者尽可能近的地方)。HBase 的数据在写的时候是本地的,但是当 region 被迁移的时候,数据就可能不再满足本地性了,直到完成 compaction,才能又恢复到本地。
  • Hadoop NameNode 维护了所有 HDFS 物理 data block 的元信息。
这可能是最容易理解的Hbase架构全解,10分钟学会,建议收藏

阅读更多